Parameter evaluation for parameterized rules (reloaded)

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Parameter evaluation for parameterized rules (reloaded)

sulfinu
Hello again, after quite some time.

After re-reading the documentation, I have tried to use Vars for passing parameter values among rules, parameters that must be evaluated when actually parsing, not when instrumenting the code. Naturally, I discovered that a rule like

Rule DYNAMIC_VALUE(final Var<Type> type)

can only be passed a proper value by using an assigning action like

return Sequence(
    .
    .
                                ACTION(type.set(obj.getValueType())),
                                DYNAMIC_VALUE(type)
    .
    .
);

But I cannot call a rule VALUE(Type), where Type is an enumeration, by simply writing

Rule DYNAMIC_VALUE(final Var<Type> type)
{
                return Sequence(
                                new Action() {
                                        @Override
                                        public boolean run(Context context)
                                        {
                                                System.out.println(">>> calling VALUE(" + type.get() + ")!");
                                                return true;
                                        }
                                },
                                VALUE(type.get())
    );
}

since the VALUE() rule gets called with a null parameter value, in spite of a correct output at System.out.

Is there a way to do it, or do I have to change the signature of VALUE() to take a Var for a parameter (and the code inside, and the calling lines)?

Also, Mathias, do you remember I asked you once upon a time which is the equality test for parameter values when choosing a cached result for a parameterized rule and you said it was "equals()" (via Map), not "=="? Well, I didn't find any overridden "equals()" in org.parboiled.support.Var or org.parboiled.common.Reference, so as to take into account the actual stored value.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Parameter evaluation for parameterized rules (reloaded)

mathias
Administrator
Sulfinu,

> But I cannot call a rule VALUE(Type), where Type is an enumeration, by simply writing
>
> Rule DYNAMIC_VALUE(final Var<Type> type)
> {
>                 return Sequence(
>                                 new Action() {
>                                         @Override
>                                         public boolean run(Context context)
>                                         {
>                                                 System.out.println(">>> calling VALUE(" + type.get() + ")!");
>                                                 return true;
>                                         }
>                                 },
>                                 VALUE(type.get())
>     );
> }
>
> since the VALUE() rule gets called with a null parameter value, in spite of a correct output at System.out.
>
> Is there a way to do it, or do I have to change the signature of VALUE() to take a Var for a parameter (and the code inside, and the calling lines)?

If I understand correctly, you are trying to pass a value that is created at runtime to a rule that uses that value at parser generation time?
This cannot work!
The reason you are seeing VALUE(Type) be called with a `null` argument is, that no Type instance is available at parser creation time.
I'm not sure what exactly it is you are trying to do, but if you want to have access to the Type instance created as parsing time you will have to use Var<Type> parameters instead of simple `Type` parameters.

> Also, Mathias, do you remember I asked you once upon a time which is the equality test for parameter values when choosing a cached result for a parameterized rule and you said it was "equals()" (via Map), not "=="? Well, I didn't find any overridden "equals()" in org.parboiled.support.Var or org.parboiled.common.Reference, so as to take into account the actual stored value.

Yes, that is true. Rule caching is something that runs during parser creation time, not during parser execution time. Therefore it wouldn't make any sense to take the stored values of Reference instances into account for caching.

Cheers,
Mathias

---
[hidden email]
http://www.parboiled.org

On 31.10.2011, at 16:20, sulfinu [via parboiled users] wrote:

> Hello again, after quite some time.
>
> After re-reading the documentation, I have tried to use Vars for passing parameter values among rules, parameters that must be evaluated when actually parsing, not when instrumenting the code. Naturally, I discovered that a rule like
>
> Rule DYNAMIC_VALUE(final Var<Type> type)
>
> can only be passed a proper value by using an assigning action like
>
> return Sequence(
>     .
>     .
>                                 ACTION(type.set(obj.getValueType())),
>                                 DYNAMIC_VALUE(type)
>     .
>     .
> );
>
> But I cannot call a rule VALUE(Type), where Type is an enumeration, by simply writing
>
> Rule DYNAMIC_VALUE(final Var<Type> type)
> {
>                 return Sequence(
>                                 new Action() {
>                                         @Override
>                                         public boolean run(Context context)
>                                         {
>                                                 System.out.println(">>> calling VALUE(" + type.get() + ")!");
>                                                 return true;
>                                         }
>                                 },
>                                 VALUE(type.get())
>     );
> }
>
> since the VALUE() rule gets called with a null parameter value, in spite of a correct output at System.out.
>
> Is there a way to do it, or do I have to change the signature of VALUE() to take a Var for a parameter (and the code inside, and the calling lines)?
>
> Also, Mathias, do you remember I asked you once upon a time which is the equality test for parameter values when choosing a cached result for a parameterized rule and you said it was "equals()" (via Map), not "=="? Well, I didn't find any overridden "equals()" in org.parboiled.support.Var or org.parboiled.common.Reference, so as to take into account the actual stored value.
>
> If you reply to this email, your message will be added to the discussion below:
> http://users.parboiled.org/Parameter-evaluation-for-parameterized-rules-reloaded-tp3468079p3468079.html
> To start a new topic under parboiled users, email [hidden email]
> To unsubscribe from parboiled users, click here.

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Parameter evaluation for parameterized rules (reloaded)

sulfinu
Well, Mathias, you might ne surprised to learn that this kind of code works, in the sense that at runtime it passes correctly the parameter from the DYNAMIC_VALUE() rule to the VALUE() rule:

Rule DYNAMIC_VALUE(Type type)
        {
                return Sequence(
                                VALUE(type),
    .
    .
  )
}

Tested thoroughly! The only problem is that one needs to invoke the parse runner on DYNAMIC_VALUE() directly, which requires a ton of workarounds.
Anyway, the reason I want to pass a Type instead of a Var<Type> to the VALUE() rule is that VALUE() is called from many other rules with a constant Type (enum value). Changing that to Var<Type> introduces an unwanted overhead.

You might also reconsider this statement, since result caching is decided during parser creation, but actually performed at runtime:
>Rule caching is something that runs during parser creation time, not during parser execution time. Therefore it wouldn't make any sense to take the stored values of Reference instances into account for caching.

Thank you for your replies.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Parameter evaluation for parameterized rules (reloaded)

mathias
Administrator
Sulfinu,

in order to be able to successfully work with parboiled one needs to understand that the following three phases are completely separated:

1. Compile time of your parboiled application
2. Runtime of your parboiled application
  a) parser creation time = execution of your Rule methods
  b) parser execution time = application of the parser rules to actual input

If you have the following code:

> Rule DYNAMIC_VALUE(Type type)
>         {
>                 return Sequence(
>                                 VALUE(type),
>     .
>     .
>   )
> }

then the actual object instance passed to the DYNAMIC_VALUE method is determined in phase 2a and not in phase 2b, i.e. it is independent of the actual parsing input because the DYNAMIC_VALUE method is called _before_ any input is being consumed!

> You might also reconsider this statement, since result caching is decided during parser creation, but actually performed at runtime:

It is performed at runtime of your application (naturally), but in phase 2a not in 2b!

Cheers,
Mathias

---
[hidden email]
http://www.parboiled.org

On 02.11.2011, at 10:50, sulfinu [via parboiled users] wrote:

> Well, Mathias, you might ne surprised to learn that this kind of code works, in the sense that at runtime it passes correctly the parameter from the DYNAMIC_VALUE() rule to the VALUE() rule:
>
> Rule DYNAMIC_VALUE(Type type)
>         {
>                 return Sequence(
>                                 VALUE(type),
>     .
>     .
>   )
> }
>
> Tested thoroughly! The only problem is that one needs to invoke the parse runner on DYNAMIC_VALUE() directly, which requires a ton of workarounds.
> Anyway, the reason I want to pass a Type instead of a Var<Type> to the VALUE() rule is that VALUE() is called from many other rules with a constant Type (enum value). Changing that to Var<Type> introduces an unwanted overhead.
>
> You might also reconsider this statement, since result caching is decided during parser creation, but actually performed at runtime:
> >Rule caching is something that runs during parser creation time, not during parser execution time. Therefore it wouldn't make any sense to take the stored values of Reference instances into account for caching.
>
> Thank you for your replies.
>
> If you reply to this email, your message will be added to the discussion below:
> http://users.parboiled.org/Parameter-evaluation-for-parameterized-rules-reloaded-tp3468079p3473314.html
> To start a new topic under parboiled users, email [hidden email]
> To unsubscribe from parboiled users, click here.

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Parameter evaluation for parameterized rules (reloaded)

sulfinu
I never ever talked in my messages about phase 1. Both here and in other conversations I only talked about the following:
2.a) "parser creation" (your terms), "rule instrumentation" (my terms)
2.b) "runtime, execution time" (your terms), "actual input parsing" (my terms)

Trust me, there's been no misunderstanding here and I'm perfectly aware as to why parameters are eagerly evaluated during 2.a). I only showed there's a case when I DO NOT use Vars and it works correctly (i.e. parameter value is not eagerly used when determining rule dependency) just because the called rule is given as actual parameter the current rule's parameter:

 Rule DYNAMIC_VALUE(Type type)
{
                return Sequence(
                                VALUE(type),
    .
    .
  )
}

My point is that IDEALLY the Parboiled instrumenter (parser generator) should not follow parameterized rule calls when building the dependency tree unless the values used in those calls are CONSTANTS. Otherwise, the use of parameterized rules is impossible, even with Vars. Really!

For example, my VALUE() method is a giant "switch" returning various other rules (remember the related bug I demonstrated?). Even if I change its signature to take a Var<Type> parameter, it will return a NONSENSE during the parser creation phase since the no Type has been stored yet in that Var. Hence, the rule tree ends up corrupted and a valid input string is wrongly rejected.

Regarding rule method result caching: are you to tell me that it is the Parboiled instrumenter that caches the result of rule methods and no caching is done during the actual parsing, when those parameters vary? Then, this is a simple continuation of the eager parameter evaluation, it makes sense...

If I remove the @Cached annotation on a parameterized method, will this prevent the Parboiled instrumenter to compute rule dependencies based on improperly initialized parameters (improperly because those values are assigned upon INPUT CONSUMING)?

I hope I made myself understood. I've found and proven the workarounds to this problem (and they are considerable in terms of programming), so I'm sure there's potential in the existing parser creation code :)
I read the documentation, I believe I proved it and I appreciate the time you invest in these discussions.
Thanks again.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Parameter evaluation for parameterized rules (reloaded)

mathias
Administrator
> My point is that IDEALLY the Parboiled instrumenter (parser generator) should not follow parameterized rule calls when building the dependency tree unless the values used in those calls are CONSTANTS. Otherwise, the use of parameterized rules is impossible, even with Vars. Really!

The parboiled "parser generator" really is no parser generator. All it does is instrumenting Rule methods with things like caching and action wrapping, nothing more. Especially it doesn't build a "dependency tree". The rule tree is created purely from executing the root rule method, which creates the root rule and all lower-level rules.
A rule method can take parameters just like any other Java method. Whether and how these variables are being used for the actual rule construction is completely up to the rule method. Parboiled does not intervene at all here.
Var<T> parameters are no different! To parboiled they are just like any other rule method parameter.
So your request that parboiled "should not follow parameterized rule calls when building the dependency tree unless the values used in those calls are CONSTANTS" doesn't really make sense to me... sorry!

Cheers,
Mathias

---
[hidden email]
http://www.parboiled.org

On 02.11.2011, at 13:58, sulfinu [via parboiled users] wrote:

> I never ever talked in my messages about phase 1. Both here and in other conversations I only talked about the following:
> 2.a) "parser creation" (your terms), "rule instrumentation" (my terms)
> 2.b) "runtime, execution time" (your terms), "actual input parsing" (my terms)
>
> Trust me, there's been no misunderstanding here and I'm perfectly aware as to why parameters are eagerly evaluated during 2.a). I only showed there's a case when I DO NOT use Vars and it works correctly (i.e. parameter value is not eagerly used when determining rule dependency) just because the called rule is given as actual parameter the current rule's parameter:
>
>  Rule DYNAMIC_VALUE(Type type)
> {
>                 return Sequence(
>                                 VALUE(type),
>     .
>     .
>   )
> }
>
> My point is that IDEALLY the Parboiled instrumenter (parser generator) should not follow parameterized rule calls when building the dependency tree unless the values used in those calls are CONSTANTS. Otherwise, the use of parameterized rules is impossible, even with Vars. Really!
>
> For example, my VALUE() method is a giant "switch" returning various other rules (remember the related bug I demonstrated?). Even if I change its signature to take a Var<Type> parameter, it will return a NONSENSE during the parser creation phase since the no Type has been stored yet in that Var. Hence, the rule tree ends up corrupted and a valid input string is wrongly rejected.
>
> Regarding rule method result caching: are you to tell me that it is the Parboiled instrumenter that caches the result of rule methods and no caching is done during the actual parsing, when those parameters vary? Then, this is a simple continuation of the eager parameter evaluation, it makes sense...
>
> If I remove the @Cached annotation on a parameterized method, will this prevent the Parboiled instrumenter to compute rule dependencies based on improperly initialized parameters (improperly because those values are assigned upon INPUT CONSUMING)?
>
> I hope I made myself understood. I've found and proven the workarounds to this problem (and they are considerable in terms of programming), so I'm sure there's potential in the existing parser creation code :)
> I read the documentation, I believe I proved it and I appreciate the time you invest in these discussions.
> Thanks again.
>
> If you reply to this email, your message will be added to the discussion below:
> http://users.parboiled.org/Parameter-evaluation-for-parameterized-rules-reloaded-tp3468079p3473715.html
> To start a new topic under parboiled users, email [hidden email]
> To unsubscribe from parboiled users, click here.

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Parameter evaluation for parameterized rules (reloaded)

sulfinu
>The rule tree is created purely from executing the root rule method, which creates the root rule and all lower-level rules.
So, it is only a constructor like XyzParseRunner(Rule) that takes a picture of the rule dependencies by calling the starting rule method, right? Should I take it that the @Cached annotation is only using during this very step and ignored afterwards?

>So your request that parboiled "should not follow parameterized rule calls when building the dependency tree unless the values used in those calls are CONSTANTS" doesn't really make sense to me... sorry!
Well then, did you at least understand that there are real life cases when parameterized rules are called with parameter values which depend on the actual parsed input? How would you address that, generally speaking? The grammar remains within PEG definition, of course, but using parametrized rules saves at some 80% of the rules declaration code.

Let's say I have to parse some properties along with their values and some modifiers:
<property_name>:<list_of_optional_modifiers, in any order>=<property_value>
Each of the modifiers has its own syntax and semantics. One modifier indicates the type of property's value (say there are 10 possible value types). The elegant solution is to have the value parsed by a parameterized rule taking as a parameter the type indicated by the corresponding modifier, instead of writing 11 complicated rules (the type modifier is optional, as are all modifiers and there's a default type for each property).

My proposal could be implemented by altering (during phase 2.a)) the calls to rule methods with non-constant parameter values and returning some open-ended results when the parse runner is created. Those results are actually materialized lazily, upon parsing, i.e. when using one of the XyzParseRunner.run() methods.

I rest my case.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Parameter evaluation for parameterized rules (reloaded)

sulfinu
What about another approach: could org.parboiled.matchers.ProxyMatcher be used in the parser definition to elude the exact rule specification and delegate the actual matching to another rule by calling ProxyMatcher.arm(Matcher) during input processing?

Thanks.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Parameter evaluation for parameterized rules (reloaded)

mathias
Administrator
Sulfinu,

this is not what the ProxyMatcher was designed for and, I think, it's not actually necessary to jump through such hoops.
If you have something really dynamic in your grammar you can simply implement a parser action that does whatever you want to do at the respective position in your grammar.
A parser action can match input, manipulate the value stack and basically do whatever it pleases.

Of course, it's probably a good idea to keep these really dynamic parts as small as possible in order to not increase overall complexity (and therefore hinder maintainability) too much...

Cheers,
Mathias

---
[hidden email]
http://www.parboiled.org

On 25.11.2011, at 11:06, sulfinu [via parboiled users] wrote:

> What about another approach: could org.parboiled.matchers.ProxyMatcher be used in the parser definition to elude the exact rule specification and delegate the actual matching to another rule by calling ProxyMatcher.arm(Matcher) during input processing?
>
> Thanks.
>
> If you reply to this email, your message will be added to the discussion below:
> http://users.parboiled.org/Parameter-evaluation-for-parameterized-rules-reloaded-tp3468079p3535727.html
> To start a new topic under parboiled users, email [hidden email]
> To unsubscribe from parboiled users, click here.
> NAML

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Parameter evaluation for parameterized rules (reloaded)

sulfinu
You are basically suggesting that a complicated solution like the one I've applied and described here
http://users.parboiled.org/ASM-Execution-can-fall-off-end-of-the-code-tp3207484p3329268.html
is less cumbersome than resorting to a possibly elegant wrapper...

I'll give it a try when I have the time.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Parameter evaluation for parameterized rules (reloaded)

sulfinu
Well, I can testify that it works like a charm via ProxyMatcher. In order to solve the problem I described above, one has to elegantly write something like this:

ProxyMatcher dynamicValue = new ProxyMatcher();
.
.
Rule DYNAMIC_VALUE(Type type)
        {
                return Sequence(
                                new Action() {
                                        @Override
                                        public boolean run(Context context)
                                        {
                                                dynamicValue.arm((Matcher) VALUE(type));
                                                return true;
                                        }
                                },
                                dynamicValue
                );
        }

The assigned value of parameter 'type' is simply null during the parser creation phase (2.a), but later on its value changes multiple times during the actual parsing (phase 2.b).
Loading...