Improve transliteration filter

[blmage]

This PR improves the transliteration filter by automatically transliterating accented variants of Latin letters and symbols from the conversion table.

Description (*)

Related Pull Requests

Fixed Issues (if relevant)

1. Fixes Magento URL Rewrites not support some characters (portuguese brazil) #29444

Manual testing scenarios (*)

1. See Magento URL Rewrites not support some characters (portuguese brazil) #29444

Questions or comments

An alternative solution would be to use UnicodeString::ascii() from symfony/string, with the following trade-offs:

• much wider range of supported symbols,
• slightly worse performance,
• additional dependency,
• behavior changes:
  • transliterated uppercase letters stay uppercase (they are currently lowercased).
  • some sequences of symbols are transliterated differently, such as these examples from the tests:
    • русский язык becomes russkij azyk instead of russkij jazyk,
    • লক্ষ্য এনালগ ওয়াচ টি ২০ becomes laksya enalaga oyaca ti 20 instead of laksoa enaalaga oyaoaca tai 20.

Contribution checklist (*)

• Pull request has a meaningful description of its purpose
• All commits are accompanied by meaningful commit messages
• All new or changed code is covered with unit/integration tests (if applicable)
• README.md files for modified modules are updated and included in the pull request if any README.md predefined sections require an update
• All automated tests passed successfully (all builds are green)

[m2-assistant]

Hi @blmage. Thank you for your contribution
Here are some useful tips how you can test your changes using Magento test environment.
Add the comment under your pull request to deploy test or vanilla Magento instance:

• @magento give me test instance - deploy test instance based on PR changes
• @magento give me 2.4-develop instance - deploy vanilla Magento instance

❗ Automated tests can be triggered manually with an appropriate comment:

• @magento run all tests - run or re-run all required tests against the PR changes
• @magento run <test-build(s)> - run or re-run specific test build(s)
  For example: @magento run Unit Tests

<test-build(s)> is a comma-separated list of build names. Allowed build names are:

1. Database Compare
2. Functional Tests CE
3. Functional Tests EE,
4. Functional Tests B2B
5. Integration Tests
6. Magento Health Index
7. Sample Data Tests CE
8. Sample Data Tests EE
9. Sample Data Tests B2B
10. Static Tests
11. Unit Tests
12. WebAPI Tests
13. Semantic Version Checker

You can find more information about the builds here

ℹ️ Please run only needed test builds instead of all when developing. Please run all test builds before sending your PR for review.

For more details, please, review the Magento Contributor Guide documentation.

⚠️ According to the Magento Contribution requirements, all Pull Requests must go through the Community Contributions Triage process. Community Contributions Triage is a public meeting.

🕙 You can find the schedule on the Magento Community Calendar page.

📞 The triage of Pull Requests happens in the queue order. If you want to speed up the delivery of your contribution, please join the Community Contributions Triage session to discuss the appropriate ticket.

🎥 You can find the recording of the previous Community Contributions Triage on the Magento Youtube Channel

✏️ Feel free to post questions/proposals/feedback related to the Community Contributions Triage process to the corresponding Slack Channel

[blmage]

@magento run all tests

[magento-automated-testing]

The requested builds are added to the queue. You should be able to see them here within a few minutes. Please re-request them if they don't show in a reasonable amount of time.

[blmage]

@magento run Functional Tests B2B, Functional Tests EE

[magento-automated-testing]

The requested builds are added to the queue. You should be able to see them here within a few minutes. Please re-request them if they don't show in a reasonable amount of time.

[Den4ik]

@magento run Functional Tests B2B

[magento-automated-testing]

The requested builds are added to the queue. You should be able to see them here within a few minutes. Please re-request them if they don't show in a reasonable amount of time.

[Den4ik]

@blmage Good job. Just trying to understood do we really need this little bit complex logic

[blmage]

Thanks! Most of the complexity comes from preserving the original behavior and restricting changes to known characters, but let me know if there is anything in particular that bothers you.