MOD-6749: Querying numeric fields using simple operators #4516

nafraf · 2024-03-14T02:45:52Z

Description

Add support to new syntax for numeric field query, simple operators: ==, !=, >, <, >=, <=
The operators are only valid using DIALECT 5.

This PR requires:
#4604 which creates the parser-v3
#4505 fix numeric range syntax

Numeric operators does not require :, but it is still required to specify a range:

@num>10
@num > 10

@num>= 10
@num >= 10

@num:[10 20]

Example:

FT.CREATE idx SCHEMA n NUMERIC
HSET doc1 n 103
HSET doc2 n 50

127.0.0.1:6379> FT.SEARCH idx '@n == 103' DIALECT 5
1) (integer) 1
2) "doc1"
3) 1) "n"
   2) "103"
127.0.0.1:6379> FT.SEARCH idx '@n!=103' DIALECT 5
1) (integer) 1
2) "doc2"
3) 1) "n"
   2) "50"
127.0.0.1:6379> FT.SEARCH idx '@n <= 103' DIALECT 5
1) (integer) 2
2) "doc1"
3) 1) "n"
   2) "103"
4) "doc2"
5) 1) "n"
   2) "50"
127.0.0.1:6379> FT.SEARCH idx '@n<100' DIALECT 5
1) (integer) 1
2) "doc2"
3) 1) "n"
   2) "50"

Which issues this PR fixes

MOD-6749

Main objects this PR modified

v3-parser
v3-lexer

Mark if applicable

This PR introduces API changes
This PR introduces serialization changes

This reverts commit 6c3ec77.

codecov · 2024-03-14T14:45:24Z

Codecov Report

Attention: Patch coverage is 87.87879% with 12 lines in your changes are missing coverage. Please review.

Project coverage is 85.97%. Comparing base (c05367b) to head (f10999a).

Files	Patch %	Lines
src/query_parser/v3/lexer.c	87.09%	12 Missing ⚠️

Additional details and impacted files

@@                    Coverage Diff                     @@
##           nafraf_parser-v3-punct    #4516      +/-   ##
==========================================================
+ Coverage                   85.94%   85.97%   +0.02%     
==========================================================
  Files                         190      190              
  Lines                       34435    34517      +82     
==========================================================
+ Hits                        29596    29676      +80     
- Misses                       4839     4841       +2

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

oshadmi

Cool! A few comments

oshadmi · 2024-03-18T18:22:27Z

src/query_parser/v2/parser.y

+
+expr(A) ::= modifier(B) COLON NOT_EQUAL param_num(C). {
+  if (C.type == QT_PARAM_NUMERIC) {
+    C.type = QT_PARAM_NUMERIC_MIN_RANGE;


For != could also be QT_PARAM_NUMERIC_MAX_RANGE?
Does it mean a parameter can be substituted with only one of -inf or +inf?

Difference between any value and -inf or +inf is always evaluated to true?

It was an error, we don't need to check the min value. Tests were added with parameters equals to +/-inf. See e0e0b4d

oshadmi · 2024-03-18T18:53:53Z

src/query_parser/v2/parser.y

+
+numeric_operator(A) ::= EQUAL EQUAL param_num(B). {
+  if (B.type == QT_PARAM_NUMERIC) {
+    B.type = QT_PARAM_NUMERIC_MIN_RANGE;


Same question for == could also be QT_PARAM_NUMERIC_MAX_RANGE?
Does it mean a parameter can be substituted with only one of -inf or +inf?

Equality of any value with -inf or +inf is never evaluated to true?

That validation was removed and test with parameters equals to +/-inf were added. See e0e0b4d

oshadmi · 2024-03-18T19:01:56Z

src/query_parser/v2/parser.y

+
+numeric_operator(A) ::= GREATER param_num(B). {
+  if (B.type == QT_PARAM_NUMERIC) {
+    B.type = QT_PARAM_NUMERIC_MIN_RANGE;


Does it mean a parameter can be substituted with only one of -inf or +inf?

Same question for the rest of the operators.

The validation was removed.

tests/pytests/test_numbers.py

oshadmi · 2024-03-18T20:02:08Z

tests/pytests/test_numbers.py

+    res1 = env.cmd('FT.SEARCH', 'idx', '@n:==-15', 'NOCONTENT')
+    env.assertEqual(res1, [0])
+
+    res1 = env.cmd('FT.SEARCH', 'idx', '@n:==-15', 'NOCONTENT')
+    env.assertEqual(res1, [0])


Duplicated?

Suggested change

res1 = env.cmd('FT.SEARCH', 'idx', '@n:==-15', 'NOCONTENT')

env.assertEqual(res1, [0])

res1 = env.cmd('FT.SEARCH', 'idx', '@n:==-15', 'NOCONTENT')

env.assertEqual(res1, [0])

res1 = env.cmd('FT.SEARCH', 'idx', '@n:==-15', 'NOCONTENT')

env.assertEqual(res1, [0])

Yes. Removed.

tests/pytests/test_numbers.py

oshadmi · 2024-03-19T15:55:50Z

tests/pytests/test.py

+    env.assertEqual(res1, expected)
+
+    if not env.isCluster():
+        # FT.EXPLAINCLI is not supported by the coordinator


Can be united with testing FT.EXPLAIN

Test united in function _testExplain()

src/query_parser/v3/lexer.rl

…raf_fix-num-range-syntax

raz-mon

Muy Rico! 🍗

raz-mon · 2024-05-09T11:47:59Z

src/query_parser/v3/parser.y

+numeric_operator(A) ::= EQUAL_EQUAL param_num(B). {
+  A = NewNumericFilterQueryParam_WithParams(ctx, &B, &B, 1, 1);
+}
+
+numeric_operator(A) ::= GT param_num(B). {
+  A = NewNumericFilterQueryParam_WithParams(ctx, &B, NULL, 0, 1);
+}
+
+numeric_operator(A) ::= GE param_num(B). {
+  A = NewNumericFilterQueryParam_WithParams(ctx, &B, NULL, 1, 1);
+}
+
+numeric_operator(A) ::= LT param_num(B). {
+  A = NewNumericFilterQueryParam_WithParams(ctx, NULL, &B, 1, 0);
+}
+
+numeric_operator(A) ::= LE param_num(B). {
+  A = NewNumericFilterQueryParam_WithParams(ctx, NULL, &B, 1, 1);
+}


#4604) * Create parser v3 * Test dialect: DEFAULT_DIALECT as module arg * More tests for text queries * Improve invalid syntax text * Fix parser v3, unary op after field name * Add missint test with modifierlist * WIP: Test isempty() with DIALECT 5 * test_v1_vs_v2_vs_v5() * Add tests to improve codecov using dialect 5 * One more test to improve codecov * Add WITHCOUNT to fix test with DIALECT 5 * Fix testEmptyValueTags() for DIALECT > 2 * Fix test_tags * Test float without leading zero * Fix wrong float number test * Test ragel minization at the end of compilation * Revert "Test ragel minization at the end of compilation" This reverts commit 0dae78b. * Fix make-parser.mk * cpp-test parser v3 * Fix float number syntax, leading zero is optional * Test number format * Support numbers with multiple signs * Create macros in parser.y v3 * Use set_max_dialect * Fix testEmptyValueTags() * MOD-6750 Fix numeric range syntax (#4505) Fix numeric range syntax * MOD-6749: Querying numeric fields using simple operators (#4516)

(cherry picked from commit 4ea8fc8)

* Add query_parser/v3 * Add more tag tests * Add test for TEXT testExact * autoescaping single tag between brackets * Fix tests COORD=1 * Fix wildcard support * wildcard + prefix/infix/suffix is invalid * Test backward compatibility * Test some uncovered cases * Test prefix/infix/suffix with TEXT field * Remove temporary debug messages * Test: escape 'w' single_tag * Add more tag tests * WIP: Tests to increase coverage parser v3 * Add more tests * Fix lexer.c v3 * Fix test invalid syntax * Test punct and cntrl characters * split unescaped_tag rule * Add one more test UNESCAPED_TAG * Fix expected test format in cluster * Update src/query_parser/v3/lexer.rl - Fix description Co-authored-by: Omer Shadmi <76992134+oshadmi@users.noreply.github.com> * Test pipe with dialect < 5, add comment about backslack escaping * Test escaping $ * One more test escaping $ * lexer v3 - remove leading and trailing spaces * Test short tags * Add JSON tests * Use comma separator for JSON tests * Add test testTagUNF() * Test tag autoescaping using DEFAULT_DIALECT 5 * More test using DEFAULT DIALECT 5 in test_search_params: test_geo, test_attr, test_binary_data. * Revert changes to test_search_params:test_geo * testTagUNF: Create index before hashes * Revert change in QueryNode_DumpSds() * Test aggregate with TAG autoescaping * Fix testTagAutoescaping, remove additional right curly brace * Update testDialect5InvalidSyntax() * update parser/v3 taking latest parser/v2 * Create parser v3 * Test dialect: DEFAULT_DIALECT as module arg * More tests for text queries * Improve invalid syntax text * Fix parser v3, unary op after field name * Add missint test with modifierlist * WIP: Test isempty() with DIALECT 5 * test_v1_vs_v2_vs_v5() * Add tests to improve codecov using dialect 5 * One more test to improve codecov * Add WITHCOUNT to fix test with DIALECT 5 * Fix testEmptyValueTags() for DIALECT > 2 * Fix test_tags * Test float without leading zero * Fix wrong float number test * Test ragel minization at the end of compilation * Revert "Test ragel minization at the end of compilation" This reverts commit 0dae78b. * Fix make-parser.mk * cpp-test parser v3 * Update parser v3 * Update lexer * Fix tests * Fix float number syntax, leading zero is optional * Test number format * Support numbers with multiple signs * Create macros in parser.y v3 * Use set_max_dialect * Fix testEmptyValueTags() * MOD-6750 Fix numeric range syntax (#4505) Fix numeric range syntax * MOD-6749: Querying numeric fields using simple operators (#4516) * Simplify single_tag * Remove unescaped_tag2 * lexer v3 - remove colon from tag expressions * Create unescaped_tag2 to create UNESCAPED_TAG without escape * Fix leading/trailing spaces deletion * Validate tok.len * Remove debugging code * Fix lexer.rl v3 format * Temp: Try to fix sanitizer * Revert "Temp: Try to fix sanitizer" This reverts commit 66002f1. * Test tag with * as literal * Fix parser: tag rules * Update tests/pytests/test_tags.py - minor typo --------- Co-authored-by: Omer Shadmi <76992134+oshadmi@users.noreply.github.com>

* Add query_parser/v3 * Add more tag tests * Add test for TEXT testExact * autoescaping single tag between brackets * Fix tests COORD=1 * Fix wildcard support * wildcard + prefix/infix/suffix is invalid * Test backward compatibility * Test some uncovered cases * Test prefix/infix/suffix with TEXT field * Remove temporary debug messages * Test: escape 'w' single_tag * Add more tag tests * WIP: Tests to increase coverage parser v3 * Add more tests * Fix lexer.c v3 * Fix test invalid syntax * Test punct and cntrl characters * split unescaped_tag rule * Add one more test UNESCAPED_TAG * Fix expected test format in cluster * Update src/query_parser/v3/lexer.rl - Fix description Co-authored-by: Omer Shadmi <76992134+oshadmi@users.noreply.github.com> * Test pipe with dialect < 5, add comment about backslack escaping * Test escaping $ * One more test escaping $ * lexer v3 - remove leading and trailing spaces * Test short tags * Add JSON tests * Use comma separator for JSON tests * Add test testTagUNF() * Test tag autoescaping using DEFAULT_DIALECT 5 * More test using DEFAULT DIALECT 5 in test_search_params: test_geo, test_attr, test_binary_data. * Revert changes to test_search_params:test_geo * testTagUNF: Create index before hashes * Revert change in QueryNode_DumpSds() * Test aggregate with TAG autoescaping * Fix testTagAutoescaping, remove additional right curly brace * Update testDialect5InvalidSyntax() * update parser/v3 taking latest parser/v2 * Create parser v3 * Test dialect: DEFAULT_DIALECT as module arg * More tests for text queries * Improve invalid syntax text * Fix parser v3, unary op after field name * Add missint test with modifierlist * WIP: Test isempty() with DIALECT 5 * test_v1_vs_v2_vs_v5() * Add tests to improve codecov using dialect 5 * One more test to improve codecov * Add WITHCOUNT to fix test with DIALECT 5 * Fix testEmptyValueTags() for DIALECT > 2 * Fix test_tags * Test float without leading zero * Fix wrong float number test * Test ragel minization at the end of compilation * Revert "Test ragel minization at the end of compilation" This reverts commit 0dae78b. * Fix make-parser.mk * cpp-test parser v3 * Update parser v3 * Update lexer * Fix tests * Fix float number syntax, leading zero is optional * Test number format * Support numbers with multiple signs * Create macros in parser.y v3 * Use set_max_dialect * Fix testEmptyValueTags() * MOD-6750 Fix numeric range syntax (#4505) Fix numeric range syntax * MOD-6749: Querying numeric fields using simple operators (#4516) * Simplify single_tag * Remove unescaped_tag2 * lexer v3 - remove colon from tag expressions * Create unescaped_tag2 to create UNESCAPED_TAG without escape * Fix leading/trailing spaces deletion * Validate tok.len * Remove debugging code * Fix lexer.rl v3 format * Temp: Try to fix sanitizer * Revert "Temp: Try to fix sanitizer" This reverts commit 66002f1. * Test tag with * as literal * Fix parser: tag rules * Update tests/pytests/test_tags.py - minor typo --------- Co-authored-by: Omer Shadmi <76992134+oshadmi@users.noreply.github.com> (cherry picked from commit e907ece)

nafraf added 14 commits March 5, 2024 12:19

Allow case unsensitive infinity, allow signed exponent

7106c33

Fail on range with 2 exclusive identical values

0556544

Allow signed params

f433c30

Allow range with 2 exclusive identical values

c20de4f

Revert breaking change: allow multiple '(' before number

e6be663

Support numbers and parameters preceded by PLUS

065b16f

Test ranges with +/-inf and parameters = +/- inf

122056c

Add FILTER tests

63cb465

Test explain numeric range

fdf458e

Test range: spaces between sign and the number are not valid

6c3ec77

Revert "Test range: spaces between sign and the number are not valid"

c7ff9d2

This reverts commit 6c3ec77.

Add numeric operators

4c4c7b0

Test invalid syntax with each operator

7b5a4dc

Rename tag field in tests

a13205a

nafraf changed the title ~~MOD-6749: Querying numeric fields use simple operators~~ MOD-6749: Querying numeric fields using simple operators Mar 14, 2024

nafraf marked this pull request as ready for review March 14, 2024 13:51

nafraf added 2 commits March 18, 2024 12:44

Merge branch 'master' into nafraf_numeric-operators

660fbeb

Merge branch 'master' into nafraf_numeric-operators

beea7df

oshadmi reviewed Mar 18, 2024

View reviewed changes

nafraf and others added 8 commits March 19, 2024 00:13

Merge branch 'master' into nafraf_fix-num-range-syntax

1cc632f

Fix and tests operators with params and +/-inf

e0e0b4d

Merge branch 'master' into nafraf_numeric-operators

b419a86

Test == and != against numeric range syntax.

0be5866

Test '>= +/-inf'

a917723

Test ft.explain and ft.explaincli

6aa6ccb

Merge branch 'master' into nafraf_numeric-operators

e2e6d2d

Fix error in parser.y during last merge

6b9b57f

oshadmi reviewed Mar 19, 2024

View reviewed changes

Fix tests on coordinator

29b3447

nafraf added 8 commits April 26, 2024 15:39

Delete test testExplainNumericRange(), tests moved to test:testExplain()

a448913

test.py clean up

3e7205a

Fix parseDoubleRange()

a515ea3

Merge branch 'nafraf_fix-num-range-syntax' into nafraf_numeric-operators

4cd559b

WIP: Allow space between PLUS and the number

160dee9

Merge branch 'nafraf_parser-v3-punct' into nafraf_fix-num-range-syntax

f84cc5f

Merge branch 'master' into nafraf_numeric-operators

c2b2633

cleanup

d359a27

nafraf marked this pull request as ready for review May 3, 2024 12:29

nafraf and others added 3 commits May 3, 2024 08:47

Merge branch 'nafraf_parser-v3-punct' into nafraf_numeric-operators

68168b1

Test invalid syntax with params

67515bc

Merge branch 'master' into nafraf_fix-num-range-syntax

4f9055d

nafraf commented May 8, 2024

View reviewed changes

src/query_parser/v3/lexer.rl Show resolved Hide resolved

nafraf and others added 6 commits May 8, 2024 11:25

Merge branch 'master' into nafraf_numeric-operators

f17f8e7

Merge remote-tracking branch 'origin/nafraf_parser-v3-punct' into naf…

deb9609

…raf_fix-num-range-syntax

Merge remote-tracking branch 'origin/nafraf_parser-v3-punct' into naf…

76eca6d

…raf_fix-num-range-syntax

Merge branch 'nafraf_parser-v3-punct' into nafraf_numeric-operators

5125a57

Merge branch 'nafraf_fix-num-range-syntax' into nafraf_numeric-operators

b9b1651

Add EQUAL EQUAL token

fad853d

raz-mon approved these changes May 9, 2024

View reviewed changes

nafraf changed the base branch from master to nafraf_parser-v3-punct May 10, 2024 18:33

nafraf requested a review from oshadmi May 12, 2024 15:32

oshadmi approved these changes May 13, 2024

View reviewed changes

Merge branch 'nafraf_parser-v3-punct' into nafraf_numeric-operators

f10999a

nafraf merged commit 4ea8fc8 into nafraf_parser-v3-punct May 15, 2024
10 checks passed

nafraf deleted the nafraf_numeric-operators branch May 15, 2024 13:39

github-actions bot pushed a commit that referenced this pull request May 16, 2024

MOD-6749: Querying numeric fields using simple operators (#4516)

e227d2f

(cherry picked from commit 4ea8fc8)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

MOD-6749: Querying numeric fields using simple operators #4516

MOD-6749: Querying numeric fields using simple operators #4516

nafraf commented Mar 14, 2024 •

edited

codecov bot commented Mar 14, 2024 •

edited

oshadmi left a comment

oshadmi Mar 18, 2024

nafraf Mar 19, 2024

oshadmi Mar 18, 2024

nafraf Mar 19, 2024 •

edited

oshadmi Mar 18, 2024

nafraf Mar 19, 2024

oshadmi Mar 18, 2024

nafraf Mar 19, 2024

oshadmi Mar 19, 2024

nafraf Mar 19, 2024

raz-mon left a comment •

edited

raz-mon May 9, 2024

MOD-6749: Querying numeric fields using simple operators #4516

MOD-6749: Querying numeric fields using simple operators #4516

Conversation

nafraf commented Mar 14, 2024 • edited

codecov bot commented Mar 14, 2024 • edited

Codecov Report

oshadmi left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nafraf Mar 19, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

raz-mon left a comment • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nafraf commented Mar 14, 2024 •

edited

codecov bot commented Mar 14, 2024 •

edited

nafraf Mar 19, 2024 •

edited

raz-mon left a comment •

edited