Support ECMAScript unicode-mode RegExp usage for 'pattern' and 'patternProperties'

[djgoku]

Instead of duplicating everything here is my problem.

If you want me to copy anything for this issue I can.

awslabs/amazon-ecs-intellisense-schema#8

I am looking to see how I can support Python Unicode regex so I can use this json schema.

[sirosen]

If I have understood the issue correctly, this is a matter of a schema using an ECMA regex syntax which python (the language) does not support.

However, luckily, we're using regress here to provide ECMA-compatible regex support for format. Which means that I need to know what type of match is failing, and then with through instrumenting the right support.

[djgoku]

Thanks for looking into this. I am opening to helping if I can. I'll try to check out the errors again tomorrow.

[sirosen]

To put all the information for this in one place (with some fun emojis):

• 🔍 The issue in question is that pattern usage needs to support unicode-mode regexps, which can contain special escape patterns
  • 📖 https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/RegExp/unicode#description
• 🤞 Will need to assess whether or not regress already implements this
• 🛠️ There is a well-defined path for plugging in an alternative implementation of pattern and patternProperties
  • 🤓 This looks interesting and doable, but will require some careful reading through jsonschema source
• ❓ I still need some nice MRE for this issue so that I can test the results
  • 🥳 that's a great place to help out if anyone has time -- create a small example of a schema with a regex which requires unicode-mode, and positive and negative cases for that schema

[Rojax]

I'm having a similar issue described here: usnistgov/metaschema#770.

In short:
Having this

"pattern": "^(\\p{L}|_)(\\p{L}|\\p{N}|[.\\-_])*$" 

results in this

Error: schemafile was not valid: '^(\\p{L}|_)(\\p{L}|\\p{N}|[.\\-_])*$' is not a 'regex'
Failed validating 'format' in metaschema['properties']['definitions']['additionalProperties']['properties']['pattern']:
    {'type': 'string', 'format': 'regex'}
On schema['definitions']['TokenDatatype']['pattern']:
    '^(\\p{L}|_)(\\p{L}|\\p{N}|[.\\-_])*$'
SchemaError: '^(\\p{L}|_)(\\p{L}|\\p{N}|[.\\-_])*$' is not a 'regex'
Failed validating 'format' in metaschema['properties']['definitions']['additionalProperties']['properties']['pattern']:
    {'type': 'string', 'format': 'regex'}
On schema['definitions']['TokenDatatype']['pattern']:
    '^(\\p{L}|_)(\\p{L}|\\p{N}|[.\\-_])*$'

[sirosen]

I took some time today to do the internal restructuring I've been meaning to do, in order to make this possible. It's most of the way there, but I've hit a bit of a strange case with custom validators which I need to sort out. And I still need to put together a good test case to verify my new work.

I think this will not work with arbitrary custom validators, at least for the initial version. In order to attach the alternate pattern validation to a validator class, I'm using the extend API. In theory, a custom validator class could have changes which that API will not preserve. I need to work out how to document this, since it's subtle.

And the other notable thing here is that this is a change to pattern but not patternProperties, at least at the moment. The two are different and each requires it's own implementation, though they can share some bits.

[sirosen]

I've just released v0.31.0, which uses unicode-mode JS regexes by default! 🎉

You can control the behavior with the (new) --regex-variant flag, e.g., --regex-variant nonunicode or --regex-variant python.
As always, I hope everyone gets benefit from the new feature and lets me know if they see any issues!

[Rojax]

Just tested v0.31.0 and can confirm that my issue described in #353 (comment) is now gone. Thank you!