From 978555653b1de15c2562cc1ea5f263c1c4fb59eb Mon Sep 17 00:00:00 2001 From: Richard Hansen Date: Sat, 20 Nov 2021 22:04:07 -0500 Subject: [PATCH 001/446] Refine `CHANGELOG.md` --- CHANGELOG.md | 120 ++++++++++++++++++++++++++------------------------- 1 file changed, 62 insertions(+), 58 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index d5d4bd07d..1004042bf 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,62 +17,64 @@ ### Compatibility changes * The `logconfig` setting is deprecated. -* For plugin authors: - * Etherpad now uses [jsdom](https://github.com/jsdom/jsdom) instead of - [cheerio](https://cheerio.js.org/) for processing HTML imports. There are - two consequences of this change: - * `require('ep_etherpad-lite/node_modules/cheerio')` no longer works. To - fix, your plugin should directly depend on `cheerio` and do - `require('cheerio')`. - * The `node` context argument passed to the `collectContentImage` hook is - now an - [`HTMLImageElement`](https://developer.mozilla.org/en-US/docs/Web/API/HTMLImageElement) - object rather than a Cheerio Node-like object, so the API is slightly - different. See - [citizenos/ep_image_upload#49](https://github.com/citizenos/ep_image_upload/pull/49) - for an example fix. - * The `clientReady` server-side hook is deprecated; use the new `userJoin` - hook instead. - * The `init_` server-side hooks are now run every time Etherpad - starts up, not just the first time after the named plugin is installed. - * The `userLeave` server-side hook's context properties have changed: - * `auth`: Deprecated. - * `author`: Deprecated; use the new `authorId` property instead. - * `readonly`: Deprecated; use the new `readOnly` property instead. - * `rev`: Deprecated. - * Changes to the `src/static/js/Changeset.js` library: - * `opIterator()`: The unused start index parameter has been removed, as has - the unused `lastIndex()` method on the returned object. - * `smartOpAssembler()`: The returned object's `appendOpWithText()` method is - deprecated without a replacement available to plugins (if you need one, - let us know and we can make the private `opsFromText()` function public). - * Several functions that should have never been public are no longer - exported: `applyZip()`, `assert()`, `clearOp()`, `cloneOp()`, `copyOp()`, - `error()`, `followAttributes()`, `opString()`, `stringOp()`, - `textLinesMutator()`, `toBaseTen()`, `toSplices()`. + +#### For plugin authors + +* Etherpad now uses [jsdom](https://github.com/jsdom/jsdom) instead of + [cheerio](https://cheerio.js.org/) for processing HTML imports. There are two + consequences of this change: + * `require('ep_etherpad-lite/node_modules/cheerio')` no longer works. To fix, + your plugin should directly depend on `cheerio` and do `require('cheerio')`. + * The `collectContentImage` hook's `node` context property is now an + [`HTMLImageElement`](https://developer.mozilla.org/en-US/docs/Web/API/HTMLImageElement) + object rather than a Cheerio Node-like object, so the API is slightly + different. See + [citizenos/ep_image_upload#49](https://github.com/citizenos/ep_image_upload/pull/49) + for an example fix. +* The `clientReady` server-side hook is deprecated; use the new `userJoin` hook + instead. +* The `init_` server-side hooks are now run every time Etherpad + starts up, not just the first time after the named plugin is installed. +* The `userLeave` server-side hook's context properties have changed: + * `auth`: Deprecated. + * `author`: Deprecated; use the new `authorId` property instead. + * `readonly`: Deprecated; use the new `readOnly` property instead. + * `rev`: Deprecated. +* Changes to the `src/static/js/Changeset.js` library: + * `opIterator()`: The unused start index parameter has been removed, as has + the unused `lastIndex()` method on the returned object. + * `smartOpAssembler()`: The returned object's `appendOpWithText()` method is + deprecated without a replacement available to plugins (if you need one, let + us know and we can make the private `opsFromText()` function public). + * Several functions that should have never been public are no longer exported: + `applyZip()`, `assert()`, `clearOp()`, `cloneOp()`, `copyOp()`, `error()`, + `followAttributes()`, `opString()`, `stringOp()`, `textLinesMutator()`, + `toBaseTen()`, `toSplices()`. ### Notable enhancements * Simplified pad reload after importing an `.etherpad` file. -* For plugin authors: - * `clientVars` was added to the context for the `postAceInit` client-side - hook. Plugins should use this instead of the `clientVars` global variable. - * New `userJoin` server-side hook. - * The `userLeave` server-side hook has a new `socket` context property. - * The `helper.aNewPad()` function (accessible to client-side tests) now - accepts hook functions to inject when opening a pad. This can be used to - test any new client-side hooks your plugin provides. - * Chat improvements: - * The `chatNewMessage` client-side hook context has new properties: - * `message`: Provides access to the raw message object so that plugins can - see the original unprocessed message text and any added metadata. - * `rendered`: Allows plugins to completely override how the message is - rendered in the UI. - * New `chatSendMessage` client-side hook that enables plugins to process the - text before sending it to the server or augment the message object with - custom metadata. - * New `chatNewMessage` server-side hook to process new chat messages before - they are saved to the database and relayed to users. + +#### For plugin authors + +* `clientVars` was added to the context for the `postAceInit` client-side hook. + Plugins should use this instead of the `clientVars` global variable. +* New `userJoin` server-side hook. +* The `userLeave` server-side hook has a new `socket` context property. +* The `helper.aNewPad()` function (accessible to client-side tests) now + accepts hook functions to inject when opening a pad. This can be used to + test any new client-side hooks your plugin provides. +* Chat improvements: + * The `chatNewMessage` client-side hook context has new properties: + * `message`: Provides access to the raw message object so that plugins can + see the original unprocessed message text and any added metadata. + * `rendered`: Allows plugins to completely override how the message is + rendered in the UI. + * New `chatSendMessage` client-side hook that enables plugins to process the + text before sending it to the server or augment the message object with + custom metadata. + * New `chatNewMessage` server-side hook to process new chat messages before + they are saved to the database and relayed to users. # 1.8.14 @@ -130,8 +132,8 @@ * Disabled wtfnode dump by default. * Send `USER_NEWINFO` messages on reconnect. * Fixed loading in a hidden iframe. -* Fixed a race condition with composition. (Thanks @ingoncalves for an exceptionally - detailed analysis and @rhansen for the fix.) +* Fixed a race condition with composition. (Thanks @ingoncalves for an + exceptionally detailed analysis and @rhansen for the fix.) # 1.8.13 @@ -158,11 +160,13 @@ # 1.8.12 -Special mention: Thanks to Sauce Labs for additional testing tunnels to help us grow! :) +Special mention: Thanks to Sauce Labs for additional testing tunnels to help us +grow! :) ### Security patches -* Fixed a regression in v1.8.11 which caused some pad names to cause Etherpad to restart. +* Fixed a regression in v1.8.11 which caused some pad names to cause Etherpad to + restart. ### Notable fixes @@ -171,8 +175,8 @@ Special mention: Thanks to Sauce Labs for additional testing tunnels to help us * Fixed a regression in v1.8.8 that caused "Uncaught TypeError: Cannot read property '0' of undefined" with some plugins (#4885) * Less warnings in server console for supported element types on import. -* Support Azure and other network share installations by using a - more truthful relative path. +* Support Azure and other network share installations by using a more truthful + relative path. ### Notable enhancements From 8274e01d34bef7d2728cd05ca4f908f5b5d727bf Mon Sep 17 00:00:00 2001 From: Richard Hansen Date: Sun, 21 Nov 2021 01:39:53 -0500 Subject: [PATCH 002/446] Add notable enhancements/fixes to 1.8.15 changelog --- CHANGELOG.md | 39 ++++++++++++++++++++++++++++++++++++++- 1 file changed, 38 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 1004042bf..777571bdd 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -13,6 +13,7 @@ your own [authentication](https://etherpad.org/doc/v1.8.14/#index_authenticate) and [authorization](https://etherpad.org/doc/v1.8.14/#index_authorize) plugins). +* Updated dependencies. ### Compatibility changes @@ -51,8 +52,42 @@ `followAttributes()`, `opString()`, `stringOp()`, `textLinesMutator()`, `toBaseTen()`, `toSplices()`. -### Notable enhancements +### Notable enhancements and fixes +* Accessibility fix for JAWS screen readers. +* Fixed "clear authorship" error (see issue #5128). +* Etherpad now considers square brackets to be valid URL characters. +* The server no longer crashes if an exception is thrown while processing a + message from a client. +* The `useMonospaceFontGlobal` setting now works (thanks @Lastpixl!). +* Chat improvements: + * The message input field is now a text area, allowing multi-line messages + (use shift-enter to insert a newline). + * Whitespace in chat messages is now preserved. +* Docker improvements: + * New `HEALTHCHECK` instruction (thanks @Gared!). + * New `settings.json` variables: `DB_COLLECTION`, `DB_URL`, + `SOCKETIO_MAX_HTTP_BUFFER_SIZE`, `DUMP_ON_UNCLEAN_EXIT` (thanks + @JustAnotherArchivist!). + * `.ep_initialized` files are no longer created. +* Worked around a [Firefox Content Security Policy + bug](https://bugzilla.mozilla.org/show_bug.cgi?id=1721296) that caused CSP + failures when `'self'` was in the CSP header. See issue #4975 for details. +* UeberDB upgraded from v1.4.10 to v1.4.18. For details, see the [ueberDB + changelog](https://github.com/ether/ueberDB/blob/master/CHANGELOG.md). + Highlights: + * The `postgrespool` driver was renamed to `postgres`, replacing the old + driver of that name. If you used the old `postgres` driver, you may see an + increase in the number of database connections. + * For `postgres`, you can now set the `dbSettings` value in `settings.json` to + a connection string (e.g., `"postgres://user:password@host/dbname"`) instead + of an object. + * For `mongodb`, the `dbName` setting was renamed to `database` (but `dbName` + still works for backwards compatibility) and is now optional (if unset, the + database name in `url` is used). +* `/admin/settings` now honors the `--settings` command-line argument. +* Fixed "Author *X* tried to submit changes as author *Y*" detection. +* Error message display improvements. * Simplified pad reload after importing an `.etherpad` file. #### For plugin authors @@ -75,6 +110,8 @@ custom metadata. * New `chatNewMessage` server-side hook to process new chat messages before they are saved to the database and relayed to users. +* Readability improvements to browser-side error stack traces. +* Added support for socket.io message acknowledgments. # 1.8.14 From 7ed980aa59518eaaed533dbcd53cad90cd6d6b13 Mon Sep 17 00:00:00 2001 From: snyk-bot Date: Sun, 21 Nov 2021 07:04:19 +0000 Subject: [PATCH 003/446] fix: upgrade rate-limiter-flexible from 2.3.1 to 2.3.2 Snyk has created this PR to upgrade rate-limiter-flexible from 2.3.1 to 2.3.2. See this package in npm: https://www.npmjs.com/package/rate-limiter-flexible See this project in Snyk: https://app.snyk.io/org/johnmclear/project/d9a12bfb-7ccd-443f-9e22-f30d339cc8c5?utm_source=github&utm_medium=referral&page=upgrade-pr --- src/package-lock.json | 6 +++--- src/package.json | 2 +- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/src/package-lock.json b/src/package-lock.json index bccb0f8c0..9efb24edc 100644 --- a/src/package-lock.json +++ b/src/package-lock.json @@ -7579,9 +7579,9 @@ "integrity": "sha512-Hrgsx+orqoygnmhFbKaHE6c296J+HTAQXoxEF6gNupROmmGJRoyzfG3ccAveqCBrwr/2yxQ5BVd/GTl5agOwSg==" }, "rate-limiter-flexible": { - "version": "2.3.1", - "resolved": "https://registry.npmjs.org/rate-limiter-flexible/-/rate-limiter-flexible-2.3.1.tgz", - "integrity": "sha512-u4Ual0ssf/RHHxK3rqKo9W2S7ulVoNdCAOrsk1gR9JLtzqg7fGw+yaCeyBAEncsL2n6XqHh/0qJk3BPDn49BjA==" + "version": "2.3.2", + "resolved": "https://registry.npmjs.org/rate-limiter-flexible/-/rate-limiter-flexible-2.3.2.tgz", + "integrity": "sha512-Q9isA+O+L5opvwB9sYAj49SYA0EA7fndVIKne0M9OSVpzaSZm3fv/9vE61B0c9A7PvLAxzeu0l/tYM2+JTi6qw==" }, "raw-body": { "version": "2.4.0", diff --git a/src/package.json b/src/package.json index 90ac13b36..5ff7673a4 100644 --- a/src/package.json +++ b/src/package.json @@ -55,7 +55,7 @@ "npm": "^6.14.15", "openapi-backend": "^4.2.0", "proxy-addr": "^2.0.7", - "rate-limiter-flexible": "^2.3.1", + "rate-limiter-flexible": "^2.3.2", "rehype": "^11.0.0", "rehype-minify-whitespace": "^4.0.5", "request": "2.88.2", From dd9814a4b8e14b79d5d09a18ae4a63b3a452a554 Mon Sep 17 00:00:00 2001 From: snyk-bot Date: Sun, 21 Nov 2021 07:04:15 +0000 Subject: [PATCH 004/446] fix: upgrade clean-css from 5.2.1 to 5.2.2 Snyk has created this PR to upgrade clean-css from 5.2.1 to 5.2.2. See this package in npm: https://www.npmjs.com/package/clean-css See this project in Snyk: https://app.snyk.io/org/johnmclear/project/d9a12bfb-7ccd-443f-9e22-f30d339cc8c5?utm_source=github&utm_medium=referral&page=upgrade-pr --- src/package-lock.json | 6 +++--- src/package.json | 2 +- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/src/package-lock.json b/src/package-lock.json index 9efb24edc..646465cfc 100644 --- a/src/package-lock.json +++ b/src/package-lock.json @@ -1266,9 +1266,9 @@ "optional": true }, "clean-css": { - "version": "5.2.1", - "resolved": "https://registry.npmjs.org/clean-css/-/clean-css-5.2.1.tgz", - "integrity": "sha512-ooQCa1/70oRfVdUUGjKpbHuxgMgm8BsDT5EBqBGvPxMoRoGXf4PNx5mMnkjzJ9Ptx4vvmDdha0QVh86QtYIk1g==", + "version": "5.2.2", + "resolved": "https://registry.npmjs.org/clean-css/-/clean-css-5.2.2.tgz", + "integrity": "sha512-/eR8ru5zyxKzpBLv9YZvMXgTSSQn7AdkMItMYynsFgGwTveCRVam9IUPFloE85B4vAIj05IuKmmEoV7/AQjT0w==", "requires": { "source-map": "~0.6.0" } diff --git a/src/package.json b/src/package.json index 5ff7673a4..34b1f90d2 100644 --- a/src/package.json +++ b/src/package.json @@ -31,7 +31,7 @@ ], "dependencies": { "async": "^3.2.1", - "clean-css": "^5.2.1", + "clean-css": "^5.2.2", "cookie-parser": "1.4.5", "cross-spawn": "^7.0.3", "ejs": "^3.1.6", From ff0f81161fd65502154164ef4d15208ab327908a Mon Sep 17 00:00:00 2001 From: snyk-bot Date: Sun, 21 Nov 2021 07:04:10 +0000 Subject: [PATCH 005/446] fix: upgrade async from 3.2.1 to 3.2.2 Snyk has created this PR to upgrade async from 3.2.1 to 3.2.2. See this package in npm: https://www.npmjs.com/package/async See this project in Snyk: https://app.snyk.io/org/johnmclear/project/d9a12bfb-7ccd-443f-9e22-f30d339cc8c5?utm_source=github&utm_medium=referral&page=upgrade-pr --- src/package-lock.json | 6 +++--- src/package.json | 2 +- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/src/package-lock.json b/src/package-lock.json index 646465cfc..4deb73f93 100644 --- a/src/package-lock.json +++ b/src/package-lock.json @@ -939,9 +939,9 @@ "dev": true }, "async": { - "version": "3.2.1", - "resolved": "https://registry.npmjs.org/async/-/async-3.2.1.tgz", - "integrity": "sha512-XdD5lRO/87udXCMC9meWdYiR+Nq6ZjUfXidViUZGu2F1MO4T3XwZ1et0hb2++BgLfhyJwy44BGB/yx80ABx8hg==" + "version": "3.2.2", + "resolved": "https://registry.npmjs.org/async/-/async-3.2.2.tgz", + "integrity": "sha512-H0E+qZaDEfx/FY4t7iLRv1W2fFI6+pyCeTw1uN20AQPiwqwM6ojPxHxdLv4z8hi2DtnW9BOckSspLucW7pIE5g==" }, "asynckit": { "version": "0.4.0", diff --git a/src/package.json b/src/package.json index 34b1f90d2..084c43ed3 100644 --- a/src/package.json +++ b/src/package.json @@ -30,7 +30,7 @@ } ], "dependencies": { - "async": "^3.2.1", + "async": "^3.2.2", "clean-css": "^5.2.2", "cookie-parser": "1.4.5", "cross-spawn": "^7.0.3", From cddd78d892b9c36419ae724470d295d62590e9c9 Mon Sep 17 00:00:00 2001 From: snyk-bot Date: Sun, 21 Nov 2021 07:04:06 +0000 Subject: [PATCH 006/446] fix: upgrade formidable from 1.2.2 to 1.2.6 Snyk has created this PR to upgrade formidable from 1.2.2 to 1.2.6. See this package in npm: https://www.npmjs.com/package/formidable See this project in Snyk: https://app.snyk.io/org/johnmclear/project/d9a12bfb-7ccd-443f-9e22-f30d339cc8c5?utm_source=github&utm_medium=referral&page=upgrade-pr --- src/package-lock.json | 6 +++--- src/package.json | 2 +- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/src/package-lock.json b/src/package-lock.json index 4deb73f93..12a7b3643 100644 --- a/src/package-lock.json +++ b/src/package-lock.json @@ -2499,9 +2499,9 @@ } }, "formidable": { - "version": "1.2.2", - "resolved": "https://registry.npmjs.org/formidable/-/formidable-1.2.2.tgz", - "integrity": "sha512-V8gLm+41I/8kguQ4/o1D3RIHRmhYFG4pnNyonvua+40rqcEmT4+V71yaZ3B457xbbgCsCfjSPi65u/W6vK1U5Q==" + "version": "1.2.6", + "resolved": "https://registry.npmjs.org/formidable/-/formidable-1.2.6.tgz", + "integrity": "sha512-KcpbcpuLNOwrEjnbpMC0gS+X8ciDoZE1kkqzat4a8vrprf+s9pKNQ/QIwWfbfs4ltgmFl3MD177SNTkve3BwGQ==" }, "forwarded": { "version": "0.2.0", diff --git a/src/package.json b/src/package.json index 084c43ed3..a468e5480 100644 --- a/src/package.json +++ b/src/package.json @@ -42,7 +42,7 @@ "express-session": "1.17.2", "fast-deep-equal": "^3.1.3", "find-root": "1.1.0", - "formidable": "1.2.2", + "formidable": "1.2.6", "http-errors": "1.8.0", "js-cookie": "^3.0.1", "jsdom": "^17.0.0", From 3ec5e8473702ffc706eae5573a527483ec397410 Mon Sep 17 00:00:00 2001 From: Richard Hansen Date: Sun, 21 Nov 2021 03:00:29 -0500 Subject: [PATCH 007/446] lint: Update ESLint dependencies --- src/package-lock.json | 109 +++++++++++++++++++----------------------- src/package.json | 6 +-- 2 files changed, 51 insertions(+), 64 deletions(-) diff --git a/src/package-lock.json b/src/package-lock.json index 12a7b3643..fd7e6d3cb 100644 --- a/src/package-lock.json +++ b/src/package-lock.json @@ -349,18 +349,18 @@ } }, "@babel/helper-validator-identifier": { - "version": "7.14.9", - "resolved": "https://registry.npmjs.org/@babel/helper-validator-identifier/-/helper-validator-identifier-7.14.9.tgz", - "integrity": "sha512-pQYxPY0UP6IHISRitNe8bsijHex4TWZXi2HwKVsjPiltzlhse2znVcm9Ace510VT1kxIHjGJCZZQBX2gJDbo0g==", + "version": "7.15.7", + "resolved": "https://registry.npmjs.org/@babel/helper-validator-identifier/-/helper-validator-identifier-7.15.7.tgz", + "integrity": "sha512-K4JvCtQqad9OY2+yTU8w+E82ywk/fe+ELNlt1G8z3bVGlZfn/hOcQQsUhGhW/N+tb3fxK800wLtKOE/aM0m72w==", "dev": true }, "@babel/highlight": { - "version": "7.14.5", - "resolved": "https://registry.npmjs.org/@babel/highlight/-/highlight-7.14.5.tgz", - "integrity": "sha512-qf9u2WFWVV0MppaL877j2dBtQIDgmidgjGk5VIMw3OadXvYaXn66U1BFlH2t4+t3i+8PhedppRv+i40ABzd+gg==", + "version": "7.16.0", + "resolved": "https://registry.npmjs.org/@babel/highlight/-/highlight-7.16.0.tgz", + "integrity": "sha512-t8MH41kUQylBtu2+4IQA3atqevA2lRgqA2wyVB/YiWmsDSuylZZuXOUy9ric30hfzauEFfdsuk/eXTRrGrfd0g==", "dev": true, "requires": { - "@babel/helper-validator-identifier": "^7.14.5", + "@babel/helper-validator-identifier": "^7.15.7", "chalk": "^2.0.0", "js-tokens": "^4.0.0" } @@ -477,9 +477,9 @@ } }, "@humanwhocodes/object-schema": { - "version": "1.2.0", - "resolved": "https://registry.npmjs.org/@humanwhocodes/object-schema/-/object-schema-1.2.0.tgz", - "integrity": "sha512-wdppn25U8z/2yiaT6YGquE6X8sSv7hNMWSXYSSU1jGv/yd6XqjXgTDJ8KP4NgjTXfJ3GbRjeeb8RTV7a/VpM+w==", + "version": "1.2.1", + "resolved": "https://registry.npmjs.org/@humanwhocodes/object-schema/-/object-schema-1.2.1.tgz", + "integrity": "sha512-ZnQMnLV4e7hDlUvw8H+U8ASL02SS2Gn6+9Ac3wGGLIe7+je2AeAOxPY+izIPJDfFDb7eDjev0Us8MO1iFRN8hA==", "dev": true }, "@js-joda/core": { @@ -1853,9 +1853,9 @@ } }, "ansi-regex": { - "version": "5.0.0", - "resolved": "https://registry.npmjs.org/ansi-regex/-/ansi-regex-5.0.0.tgz", - "integrity": "sha512-bY6fj56OUQ0hU1KjFNDQuJFezqKdrAyFdIevADiqrWHwSlbmBNMHp5ak2f40Pm8JTFyM2mqxkG6ngkHO11f/lg==", + "version": "5.0.1", + "resolved": "https://registry.npmjs.org/ansi-regex/-/ansi-regex-5.0.1.tgz", + "integrity": "sha512-quJQXlTSUGL2LH9SUXo8VwsY4soanhgo6LNSm84E1LBcE8s3O0wpdiRzyR9z/ZZJMlMWv37qOOb9pdJlMUEKFQ==", "dev": true }, "ansi-styles": { @@ -1981,12 +1981,12 @@ "dev": true }, "strip-ansi": { - "version": "6.0.0", - "resolved": "https://registry.npmjs.org/strip-ansi/-/strip-ansi-6.0.0.tgz", - "integrity": "sha512-AuvKTrTfQNYNIctbR1K/YGTR1756GycPsg7b9bdV9Duqur4gv6aKqHXah67Z8ImS7WEz5QVcOtlfW2rZEugt6w==", + "version": "6.0.1", + "resolved": "https://registry.npmjs.org/strip-ansi/-/strip-ansi-6.0.1.tgz", + "integrity": "sha512-Y38VPSHcqkFrCpFnQ9vuSXmquuv5oXOKpGeT6aGrr3o3Gc9AlVa6JBfUSOCnbxGGZF+/0ooI7KrPuUSztUdU5A==", "dev": true, "requires": { - "ansi-regex": "^5.0.0" + "ansi-regex": "^5.0.1" } }, "strip-json-comments": { @@ -2016,15 +2016,15 @@ } }, "eslint-config-etherpad": { - "version": "2.0.0", - "resolved": "https://registry.npmjs.org/eslint-config-etherpad/-/eslint-config-etherpad-2.0.0.tgz", - "integrity": "sha512-ejBTLZiXkreSHNsdHWk/vCRkieYb6CpVZb/DH2QKbYktqRN/EFgaSISLb/8n8HZA5XvLVLbRDvDyBc/h3tIEcA==", + "version": "2.0.2", + "resolved": "https://registry.npmjs.org/eslint-config-etherpad/-/eslint-config-etherpad-2.0.2.tgz", + "integrity": "sha512-UT9QJuRS+yf/ERkxyrTsKl8CNzIGW042WXALQhO+whEMBcahEhjzME8fdoAnkJWpJVV3VJxNY/n8rjDgppYl2A==", "dev": true }, "eslint-plugin-cypress": { - "version": "2.11.3", - "resolved": "https://registry.npmjs.org/eslint-plugin-cypress/-/eslint-plugin-cypress-2.11.3.tgz", - "integrity": "sha512-hOoAid+XNFtpvOzZSNWP5LDrQBEJwbZwjib4XJ1KcRYKjeVj0mAmPmucG4Egli4j/aruv+Ow/acacoloWWCl9Q==", + "version": "2.12.1", + "resolved": "https://registry.npmjs.org/eslint-plugin-cypress/-/eslint-plugin-cypress-2.12.1.tgz", + "integrity": "sha512-c2W/uPADl5kospNDihgiLc7n87t5XhUbFDoTl6CfVkmG+kDAb5Ux10V9PoLPu9N+r7znpc+iQlcmAqT1A/89HA==", "dev": true, "requires": { "globals": "^11.12.0" @@ -2122,9 +2122,9 @@ "dev": true }, "eslint-plugin-promise": { - "version": "5.1.0", - "resolved": "https://registry.npmjs.org/eslint-plugin-promise/-/eslint-plugin-promise-5.1.0.tgz", - "integrity": "sha512-NGmI6BH5L12pl7ScQHbg7tvtk4wPxxj8yPHH47NvSmMtFneC077PSeY3huFj06ZWZvtbfxSPt3RuOQD5XcR4ng==", + "version": "5.1.1", + "resolved": "https://registry.npmjs.org/eslint-plugin-promise/-/eslint-plugin-promise-5.1.1.tgz", + "integrity": "sha512-XgdcdyNzHfmlQyweOPTxmc7pIsS6dE4MvwhXWMQ2Dxs1XAL2GJDilUsjWen6TWik0aSI+zD/PqocZBblcm9rdA==", "dev": true }, "eslint-plugin-you-dont-need-lodash-underscore": { @@ -2459,23 +2459,12 @@ "requires": { "flatted": "^3.1.0", "rimraf": "^3.0.2" - }, - "dependencies": { - "rimraf": { - "version": "3.0.2", - "resolved": "https://registry.npmjs.org/rimraf/-/rimraf-3.0.2.tgz", - "integrity": "sha512-JZkJMZkAGFFPP2YqXZXPbMlMBgsxzE8ILs4lMIX/2o0L9UBw9O/Y3o6wFw/i9YLapcUJWwqbi3kdxIPdC62TIA==", - "dev": true, - "requires": { - "glob": "^7.1.3" - } - } } }, "flatted": { - "version": "3.2.2", - "resolved": "https://registry.npmjs.org/flatted/-/flatted-3.2.2.tgz", - "integrity": "sha512-JaTY/wtrcSyvXJl4IMFHPKyFur1sE9AUqc0QnhOaJ0CxHtAoIV8pYDzeEfAaNEtGkOfq4gr3LBFmdXW5mOQFnA==", + "version": "3.2.4", + "resolved": "https://registry.npmjs.org/flatted/-/flatted-3.2.4.tgz", + "integrity": "sha512-8/sOawo8tJ4QOBX8YlQBMxL8+RLZfxMQOif9o0KUKTNTjMYElWPE0r/m5VNFxTRd0NSw8qSy8dajrwX4RYI1Hw==", "dev": true }, "follow-redirects": { @@ -2620,9 +2609,9 @@ } }, "globals": { - "version": "13.11.0", - "resolved": "https://registry.npmjs.org/globals/-/globals-13.11.0.tgz", - "integrity": "sha512-08/xrJ7wQjK9kkkRoI3OFUBbLx4f+6x3SGwcPvQ0QH6goFDrOU2oyAWrmh3dJezu65buo+HBMzAMQy6rovVC3g==", + "version": "13.12.0", + "resolved": "https://registry.npmjs.org/globals/-/globals-13.12.0.tgz", + "integrity": "sha512-uS8X6lSKN2JumVoXrbUz+uG4BYG+eiawqm3qFcT7ammfbUHeCBoJMlHcec/S3krSk73/AE/f0szYFmgAA3kYZg==", "dev": true, "requires": { "type-fest": "^0.20.2" @@ -7814,7 +7803,6 @@ "version": "3.0.2", "resolved": "https://registry.npmjs.org/rimraf/-/rimraf-3.0.2.tgz", "integrity": "sha512-JZkJMZkAGFFPP2YqXZXPbMlMBgsxzE8ILs4lMIX/2o0L9UBw9O/Y3o6wFw/i9YLapcUJWwqbi3kdxIPdC62TIA==", - "optional": true, "requires": { "glob": "^7.1.3" } @@ -8500,23 +8488,22 @@ "integrity": "sha512-9QNk5KwDF+Bvz+PyObkmSYjI5ksVUYtjW7AU22r2NKcfLJcXp96hkDWU3+XndOsUb+AQ9QhfzfCT2O+CNWT5Tw==" }, "table": { - "version": "6.7.1", - "resolved": "https://registry.npmjs.org/table/-/table-6.7.1.tgz", - "integrity": "sha512-ZGum47Yi6KOOFDE8m223td53ath2enHcYLgOCjGr5ngu8bdIARQk6mN/wRMv4yMRcHnCSnHbCEha4sobQx5yWg==", + "version": "6.7.3", + "resolved": "https://registry.npmjs.org/table/-/table-6.7.3.tgz", + "integrity": "sha512-5DkIxeA7XERBqMwJq0aHZOdMadBx4e6eDoFRuyT5VR82J0Ycg2DwM6GfA/EQAhJ+toRTaS1lIdSQCqgrmhPnlw==", "dev": true, "requires": { "ajv": "^8.0.1", - "lodash.clonedeep": "^4.5.0", "lodash.truncate": "^4.4.2", "slice-ansi": "^4.0.0", - "string-width": "^4.2.0", - "strip-ansi": "^6.0.0" + "string-width": "^4.2.3", + "strip-ansi": "^6.0.1" }, "dependencies": { "ansi-regex": { - "version": "5.0.0", - "resolved": "https://registry.npmjs.org/ansi-regex/-/ansi-regex-5.0.0.tgz", - "integrity": "sha512-bY6fj56OUQ0hU1KjFNDQuJFezqKdrAyFdIevADiqrWHwSlbmBNMHp5ak2f40Pm8JTFyM2mqxkG6ngkHO11f/lg==", + "version": "5.0.1", + "resolved": "https://registry.npmjs.org/ansi-regex/-/ansi-regex-5.0.1.tgz", + "integrity": "sha512-quJQXlTSUGL2LH9SUXo8VwsY4soanhgo6LNSm84E1LBcE8s3O0wpdiRzyR9z/ZZJMlMWv37qOOb9pdJlMUEKFQ==", "dev": true }, "is-fullwidth-code-point": { @@ -8526,23 +8513,23 @@ "dev": true }, "string-width": { - "version": "4.2.2", - "resolved": "https://registry.npmjs.org/string-width/-/string-width-4.2.2.tgz", - "integrity": "sha512-XBJbT3N4JhVumXE0eoLU9DCjcaF92KLNqTmFCnG1pf8duUxFGwtP6AD6nkjw9a3IdiRtL3E2w3JDiE/xi3vOeA==", + "version": "4.2.3", + "resolved": "https://registry.npmjs.org/string-width/-/string-width-4.2.3.tgz", + "integrity": "sha512-wKyQRQpjJ0sIp62ErSZdGsjMJWsap5oRNihHhu6G7JVO/9jIB6UyevL+tXuOqrng8j/cxKTWyWUwvSTriiZz/g==", "dev": true, "requires": { "emoji-regex": "^8.0.0", "is-fullwidth-code-point": "^3.0.0", - "strip-ansi": "^6.0.0" + "strip-ansi": "^6.0.1" } }, "strip-ansi": { - "version": "6.0.0", - "resolved": "https://registry.npmjs.org/strip-ansi/-/strip-ansi-6.0.0.tgz", - "integrity": "sha512-AuvKTrTfQNYNIctbR1K/YGTR1756GycPsg7b9bdV9Duqur4gv6aKqHXah67Z8ImS7WEz5QVcOtlfW2rZEugt6w==", + "version": "6.0.1", + "resolved": "https://registry.npmjs.org/strip-ansi/-/strip-ansi-6.0.1.tgz", + "integrity": "sha512-Y38VPSHcqkFrCpFnQ9vuSXmquuv5oXOKpGeT6aGrr3o3Gc9AlVa6JBfUSOCnbxGGZF+/0ooI7KrPuUSztUdU5A==", "dev": true, "requires": { - "ansi-regex": "^5.0.0" + "ansi-regex": "^5.0.1" } } } diff --git a/src/package.json b/src/package.json index a468e5480..1c7dba7de 100644 --- a/src/package.json +++ b/src/package.json @@ -77,13 +77,13 @@ }, "devDependencies": { "eslint": "^7.32.0", - "eslint-config-etherpad": "^2.0.0", - "eslint-plugin-cypress": "^2.11.3", + "eslint-config-etherpad": "^2.0.2", + "eslint-plugin-cypress": "^2.12.1", "eslint-plugin-eslint-comments": "^3.2.0", "eslint-plugin-mocha": "^9.0.0", "eslint-plugin-node": "^11.1.0", "eslint-plugin-prefer-arrow": "^1.2.3", - "eslint-plugin-promise": "^5.1.0", + "eslint-plugin-promise": "^5.1.1", "eslint-plugin-you-dont-need-lodash-underscore": "^6.12.0", "etherpad-cli-client": "^0.1.12", "mocha": "^9.1.1", From d0e74ada2f0856f2b0cd48c85c2f8fad2818e95c Mon Sep 17 00:00:00 2001 From: Richard Hansen Date: Tue, 16 Nov 2021 18:08:06 -0500 Subject: [PATCH 008/446] changesettracker: Remove unnecessary `.numToAttrib` check --- src/static/js/changesettracker.js | 69 +++++++++++++++---------------- 1 file changed, 34 insertions(+), 35 deletions(-) diff --git a/src/static/js/changesettracker.js b/src/static/js/changesettracker.js index 94bc5071b..0a0ef9332 100644 --- a/src/static/js/changesettracker.js +++ b/src/static/js/changesettracker.js @@ -142,43 +142,42 @@ const makeChangesetTracker = (scheduler, apool, aceCallbacksProvider) => { // Sanitize authorship // We need to replace all author attribs with thisSession.author, // in case they copy/pasted or otherwise inserted other peoples changes - if (apool.numToAttrib) { - let authorAttr; - for (const attr in apool.numToAttrib) { - if (apool.numToAttrib[attr][0] === 'author' && - apool.numToAttrib[attr][1] === authorId) { - authorAttr = Number(attr).toString(36); - } + let authorAttr; + for (const attr in apool.numToAttrib) { + if (apool.numToAttrib[attr][0] === 'author' && + apool.numToAttrib[attr][1] === authorId) { + authorAttr = Number(attr).toString(36); } - - // Replace all added 'author' attribs with the value of the current user - const cs = Changeset.unpack(userChangeset); - const iterator = Changeset.opIterator(cs.ops); - let op; - const assem = Changeset.mergingOpAssembler(); - - while (iterator.hasNext()) { - op = iterator.next(); - if (op.opcode === '+') { - let newAttrs = ''; - - op.attribs.split('*').forEach((attrNum) => { - if (!attrNum) return; - const attr = apool.getAttrib(parseInt(attrNum, 36)); - if (!attr) return; - if ('author' === attr[0]) { - // replace that author with the current one - newAttrs += `*${authorAttr}`; - } else { newAttrs += `*${attrNum}`; } // overtake all other attribs as is - }); - op.attribs = newAttrs; - } - assem.append(op); - } - assem.endDocument(); - userChangeset = Changeset.pack(cs.oldLen, cs.newLen, assem.toString(), cs.charBank); - Changeset.checkRep(userChangeset); } + + // Replace all added 'author' attribs with the value of the current user + const cs = Changeset.unpack(userChangeset); + const iterator = Changeset.opIterator(cs.ops); + let op; + const assem = Changeset.mergingOpAssembler(); + + while (iterator.hasNext()) { + op = iterator.next(); + if (op.opcode === '+') { + let newAttrs = ''; + + op.attribs.split('*').forEach((attrNum) => { + if (!attrNum) return; + const attr = apool.getAttrib(parseInt(attrNum, 36)); + if (!attr) return; + if ('author' === attr[0]) { + // replace that author with the current one + newAttrs += `*${authorAttr}`; + } else { newAttrs += `*${attrNum}`; } // overtake all other attribs as is + }); + op.attribs = newAttrs; + } + assem.append(op); + } + assem.endDocument(); + userChangeset = Changeset.pack(cs.oldLen, cs.newLen, assem.toString(), cs.charBank); + Changeset.checkRep(userChangeset); + if (Changeset.isIdentity(userChangeset)) toSubmit = null; else toSubmit = userChangeset; } From 93abc31936d29c724eaa2c004ccdaaec24ea5d9d Mon Sep 17 00:00:00 2001 From: Richard Hansen Date: Tue, 16 Nov 2021 18:16:47 -0500 Subject: [PATCH 009/446] changesettracker: Fix author attribute ID fetch --- src/static/js/changesettracker.js | 15 +++------------ 1 file changed, 3 insertions(+), 12 deletions(-) diff --git a/src/static/js/changesettracker.js b/src/static/js/changesettracker.js index 0a0ef9332..cfb6c88a3 100644 --- a/src/static/js/changesettracker.js +++ b/src/static/js/changesettracker.js @@ -139,18 +139,9 @@ const makeChangesetTracker = (scheduler, apool, aceCallbacksProvider) => { // Get my authorID const authorId = parent.parent.pad.myUserInfo.userId; - // Sanitize authorship - // We need to replace all author attribs with thisSession.author, - // in case they copy/pasted or otherwise inserted other peoples changes - let authorAttr; - for (const attr in apool.numToAttrib) { - if (apool.numToAttrib[attr][0] === 'author' && - apool.numToAttrib[attr][1] === authorId) { - authorAttr = Number(attr).toString(36); - } - } - - // Replace all added 'author' attribs with the value of the current user + // Sanitize authorship: Replace all author attributes with this user's author ID in case the + // text was copied from another author. + const authorAttr = Changeset.numToString(apool.putAttrib(['author', authorId])); const cs = Changeset.unpack(userChangeset); const iterator = Changeset.opIterator(cs.ops); let op; From cdad5c33259338fecd43439e7fd2eb1cdcc0cf99 Mon Sep 17 00:00:00 2001 From: Richard Hansen Date: Thu, 18 Nov 2021 20:52:23 -0500 Subject: [PATCH 010/446] docs: Improve `getLineHTMLForExport` server-side hook docs --- doc/api/hooks_server-side.md | 49 +++++++++++++++++------------------- 1 file changed, 23 insertions(+), 26 deletions(-) diff --git a/doc/api/hooks_server-side.md b/doc/api/hooks_server-side.md index 5e8832fd4..476c3d050 100644 --- a/doc/api/hooks_server-side.md +++ b/doc/api/hooks_server-side.md @@ -646,39 +646,36 @@ exports.clientVars = (hookName, context, callback) => { }; ``` -## getLineHTMLForExport -Called from: src/node/utils/ExportHtml.js +## `getLineHTMLForExport` -Things in context: +Called from: `src/node/utils/ExportHtml.js` -1. apool - pool object -2. attribLine - line attributes -3. text - line text +This hook will allow a plug-in developer to re-write each line when exporting to +HTML. -This hook will allow a plug-in developer to re-write each line when exporting to HTML. +Context properties: + +* `apool`: Pool object. +* `attribLine`: Line attributes. +* `line`: +* `lineContent`: +* `text`: Line text. +* `padId`: Writable (not read-only) pad identifier. Example: -``` -var Changeset = require("ep_etherpad-lite/static/js/Changeset"); -exports.getLineHTMLForExport = function (hook, context) { - var header = _analyzeLine(context.attribLine, context.apool); - if (header) { - return "<" + header + ">" + context.lineContent + ""; - } -} +```javascript +const Changeset = require('ep_etherpad-lite/static/js/Changeset'); -function _analyzeLine(alineAttrs, apool) { - var header = null; - if (alineAttrs) { - var opIter = Changeset.opIterator(alineAttrs); - if (opIter.hasNext()) { - var op = opIter.next(); - header = Changeset.opAttributeValue(op, 'heading', apool); - } - } - return header; -} +exports.getLineHTMLForExport = async (hookName, context) => { + if (!context.attribLine) return; + const opIter = Changeset.opIterator(context.attribLine); + if (!opIter.hasNext()) return; + const op = opIter.next(); + const heading = Changeset.opAttributeValue(op, 'heading', apool); + if (!heading) return; + context.lineContent = `<${heading}>${context.lineContent}`; +}; ``` ## exportHTMLAdditionalContent From 9e7b142bb7c631e599987651e5bbeda607b35444 Mon Sep 17 00:00:00 2001 From: Richard Hansen Date: Thu, 18 Nov 2021 18:59:17 -0500 Subject: [PATCH 011/446] Invert conditions to improve readability --- src/static/js/AttributeManager.js | 59 ++++++++++++------------------- src/static/js/broadcast.js | 14 +++----- src/static/js/linestylefilter.js | 38 ++++++++++---------- 3 files changed, 46 insertions(+), 65 deletions(-) diff --git a/src/static/js/AttributeManager.js b/src/static/js/AttributeManager.js index 124434031..d5b650bd5 100644 --- a/src/static/js/AttributeManager.js +++ b/src/static/js/AttributeManager.js @@ -147,13 +147,10 @@ AttributeManager.prototype = _(AttributeManager.prototype).extend({ getAttributeOnLine(lineNum, attributeName) { // get `attributeName` attribute of first char of line const aline = this.rep.alines[lineNum]; - if (aline) { - const opIter = Changeset.opIterator(aline); - if (opIter.hasNext()) { - return Changeset.opAttributeValue(opIter.next(), attributeName, this.rep.apool) || ''; - } - } - return ''; + if (!aline) return ''; + const opIter = Changeset.opIterator(aline); + if (!opIter.hasNext()) return ''; + return Changeset.opAttributeValue(opIter.next(), attributeName, this.rep.apool) || ''; }, /* @@ -163,21 +160,16 @@ AttributeManager.prototype = _(AttributeManager.prototype).extend({ getAttributesOnLine(lineNum) { // get attributes of first char of line const aline = this.rep.alines[lineNum]; + if (!aline) return []; + const opIter = Changeset.opIterator(aline); + if (!opIter.hasNext()) return []; + const op = opIter.next(); + if (!op.attribs) return []; const attributes = []; - if (aline) { - const opIter = Changeset.opIterator(aline); - let op; - if (opIter.hasNext()) { - op = opIter.next(); - if (!op.attribs) return []; - - Changeset.eachAttribNumber(op.attribs, (n) => { - attributes.push([this.rep.apool.getAttribKey(n), this.rep.apool.getAttribValue(n)]); - }); - return attributes; - } - } - return []; + Changeset.eachAttribNumber(op.attribs, (n) => { + attributes.push([this.rep.apool.getAttribKey(n), this.rep.apool.getAttribValue(n)]); + }); + return attributes; }, /* @@ -278,27 +270,22 @@ AttributeManager.prototype = _(AttributeManager.prototype).extend({ // we need to sum up how much characters each operations take until the wanted position let currentPointer = 0; - const attributes = []; let currentOperation; while (opIter.hasNext()) { currentOperation = opIter.next(); currentPointer += currentOperation.chars; - - if (currentPointer > column) { - // we got the operation of the wanted position, now collect all its attributes - Changeset.eachAttribNumber(currentOperation.attribs, (n) => { - attributes.push([ - this.rep.apool.getAttribKey(n), - this.rep.apool.getAttribValue(n), - ]); - }); - - // skip the loop - return attributes; - } + if (currentPointer <= column) continue; + const attributes = []; + Changeset.eachAttribNumber(currentOperation.attribs, (n) => { + attributes.push([ + this.rep.apool.getAttribKey(n), + this.rep.apool.getAttribValue(n), + ]); + }); + return attributes; } - return attributes; + return []; }, /* diff --git a/src/static/js/broadcast.js b/src/static/js/broadcast.js index 6ba2ef0ab..c778f6909 100644 --- a/src/static/js/broadcast.js +++ b/src/static/js/broadcast.js @@ -119,15 +119,11 @@ const loadBroadcastJS = (socket, sendSocketMsg, fireWhenAllScriptsAreLoaded, Bro const alines = this.alines; for (let i = 0; i < alines.length; i++) { Changeset.eachAttribNumber(alines[i], (n) => { - if (!seenNums[n]) { - seenNums[n] = true; - if (this.apool.getAttribKey(n) === 'author') { - const a = this.apool.getAttribValue(n); - if (a) { - authors.push(a); - } - } - } + if (seenNums[n]) return; + seenNums[n] = true; + if (this.apool.getAttribKey(n) !== 'author') return; + const a = this.apool.getAttribValue(n); + if (a) authors.push(a); }); } authors.sort(); diff --git a/src/static/js/linestylefilter.js b/src/static/js/linestylefilter.js index 84668ea46..c0a81e9da 100644 --- a/src/static/js/linestylefilter.js +++ b/src/static/js/linestylefilter.js @@ -77,26 +77,24 @@ linestylefilter.getLineStyleFilter = (lineLength, aline, textAndClassFunc, apool Changeset.eachAttribNumber(attribs, (n) => { // Give us this attributes key const key = apool.getAttribKey(n); - if (key) { - const value = apool.getAttribValue(n); - if (value) { - if (!isLineAttribMarker && AttributeManager.lineAttributes.indexOf(key) >= 0) { - isLineAttribMarker = true; - } - if (key === 'author') { - classes += ` ${linestylefilter.getAuthorClassName(value)}`; - } else if (key === 'list') { - classes += ` list:${value}`; - } else if (key === 'start') { - // Needed to introduce the correct Ordered list item start number on import - classes += ` start:${value}`; - } else if (linestylefilter.ATTRIB_CLASSES[key]) { - classes += ` ${linestylefilter.ATTRIB_CLASSES[key]}`; - } else { - const results = hooks.callAll('aceAttribsToClasses', {linestylefilter, key, value}); - classes += ` ${results.join(' ')}`; - } - } + if (!key) return; + const value = apool.getAttribValue(n); + if (!value) return; + if (!isLineAttribMarker && AttributeManager.lineAttributes.indexOf(key) >= 0) { + isLineAttribMarker = true; + } + if (key === 'author') { + classes += ` ${linestylefilter.getAuthorClassName(value)}`; + } else if (key === 'list') { + classes += ` list:${value}`; + } else if (key === 'start') { + // Needed to introduce the correct Ordered list item start number on import + classes += ` start:${value}`; + } else if (linestylefilter.ATTRIB_CLASSES[key]) { + classes += ` ${linestylefilter.ATTRIB_CLASSES[key]}`; + } else { + const results = hooks.callAll('aceAttribsToClasses', {linestylefilter, key, value}); + classes += ` ${results.join(' ')}`; } }); From 982d8ad0f2a4b209f94ca2690233b7c4186afe72 Mon Sep 17 00:00:00 2001 From: Richard Hansen Date: Fri, 19 Nov 2021 01:08:21 -0500 Subject: [PATCH 012/446] Changeset: Refactor `makeAttribsString` for readability --- src/static/js/Changeset.js | 35 +++++++++++++++++------------------ 1 file changed, 17 insertions(+), 18 deletions(-) diff --git a/src/static/js/Changeset.js b/src/static/js/Changeset.js index a335f7ccb..71d7fe115 100644 --- a/src/static/js/Changeset.js +++ b/src/static/js/Changeset.js @@ -1905,25 +1905,24 @@ exports.builder = (oldLen) => { return self; }; +/** + * Constructs an attribute string from a sequence of attributes. + * + * @param {string} opcode - The opcode for the Op that will get the resulting attribute string. + * @param {?(Attribute[]|AttributeString)} attribs - The attributes to insert into the pool + * (if necessary) and encode. If an attribute string, no checking is performed to ensure that + * the attributes exist in the pool, are in the canonical order, and contain no duplicate keys. + * If this is an iterable of attributes, `pool` must be non-null. + * @param {AttributePool} pool - Attribute pool. Required if `attribs` is an iterable of attributes, + * ignored if `attribs` is an attribute string. + * @returns {AttributeString} + */ exports.makeAttribsString = (opcode, attribs, pool) => { - // makeAttribsString(opcode, '*3') or makeAttribsString(opcode, [['foo','bar']], myPool) work - if (!attribs) { - return ''; - } else if ((typeof attribs) === 'string') { - return attribs; - } else if (pool && attribs.length) { - if (attribs.length > 1) { - attribs = attribs.slice(); - sortAttribs(attribs); - } - const result = []; - for (const pair of attribs) { - if (opcode === '=' || (opcode === '+' && pair[1])) { - result.push(`*${exports.numToString(pool.putAttrib(pair))}`); - } - } - return result.join(''); - } + if (!attribs || !['=', '+'].includes(opcode)) return ''; + if (typeof attribs === 'string') return attribs; + return sortAttribs(attribs.filter(([k, v]) => v || opcode === '=')) + .map((a) => `*${exports.numToString(pool.putAttrib(a))}`) + .join(''); }; /** From 6cf2055199afb4049b62e1bae4960f7723e8cfdd Mon Sep 17 00:00:00 2001 From: Richard Hansen Date: Wed, 17 Nov 2021 16:27:05 -0500 Subject: [PATCH 013/446] Changeset: New API to simplify attribute processing --- CHANGELOG.md | 10 + src/node/utils/tar.json | 4 + src/static/js/AttributeMap.js | 91 ++++++ src/static/js/attributes.js | 130 +++++++++ src/tests/frontend/specs/AttributeMap.js | 178 ++++++++++++ src/tests/frontend/specs/attributes.js | 343 +++++++++++++++++++++++ 6 files changed, 756 insertions(+) create mode 100644 src/static/js/AttributeMap.js create mode 100644 src/static/js/attributes.js create mode 100644 src/tests/frontend/specs/AttributeMap.js create mode 100644 src/tests/frontend/specs/attributes.js diff --git a/CHANGELOG.md b/CHANGELOG.md index 777571bdd..8850e4036 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,3 +1,13 @@ +# 1.9.0 (not yet released) + +### Notable enhancements + +#### For plugin authors + +* New APIs for processing attributes: `ep_etherpad-lite/static/js/attributes` + (low-level API) and `ep_etherpad-lite/static/js/AttributeMap` (high-level + API). + # 1.8.15 ### Security fixes diff --git a/src/node/utils/tar.json b/src/node/utils/tar.json index 896913ffe..08ae93f6b 100644 --- a/src/node/utils/tar.json +++ b/src/node/utils/tar.json @@ -54,6 +54,8 @@ , "broadcast_revisions.js" , "socketio.js" , "AttributeManager.js" + , "AttributeMap.js" + , "attributes.js" , "ChangesetUtils.js" ] , "ace2_inner.js": [ @@ -71,6 +73,8 @@ , "linestylefilter.js" , "domline.js" , "AttributeManager.js" + , "AttributeMap.js" + , "attributes.js" , "scroll.js" , "caretPosition.js" , "pad_utils.js" diff --git a/src/static/js/AttributeMap.js b/src/static/js/AttributeMap.js new file mode 100644 index 000000000..55640eb8b --- /dev/null +++ b/src/static/js/AttributeMap.js @@ -0,0 +1,91 @@ +'use strict'; + +const attributes = require('./attributes'); + +/** + * A `[key, value]` pair of strings describing a text attribute. + * + * @typedef {[string, string]} Attribute + */ + +/** + * A concatenated sequence of zero or more attribute identifiers, each one represented by an + * asterisk followed by a base-36 encoded attribute number. + * + * Examples: '', '*0', '*3*j*z*1q' + * + * @typedef {string} AttributeString + */ + +/** + * Convenience class to convert an Op's attribute string to/from a Map of key, value pairs. + */ +class AttributeMap extends Map { + /** + * Converts an attribute string into an AttributeMap. + * + * @param {AttributeString} str - The attribute string to convert into an AttributeMap. + * @param {AttributePool} pool - Attribute pool. + * @returns {AttributeMap} + */ + static fromString(str, pool) { + return new AttributeMap(pool).updateFromString(str); + } + + /** + * @param {AttributePool} pool - Attribute pool. + */ + constructor(pool) { + super(); + /** @public */ + this.pool = pool; + } + + /** + * @param {string} k - Attribute name. + * @param {string} v - Attribute value. + * @returns {AttributeMap} `this` (for chaining). + */ + set(k, v) { + k = k == null ? '' : String(k); + v = v == null ? '' : String(v); + this.pool.putAttrib([k, v]); + return super.set(k, v); + } + + toString() { + return attributes.attribsToString(attributes.sort([...this]), this.pool); + } + + /** + * @param {Iterable} entries - [key, value] pairs to insert into this map. + * @param {boolean} [emptyValueIsDelete] - If true and an entry's value is the empty string, the + * key is removed from this map (if present). + * @returns {AttributeMap} `this` (for chaining). + */ + update(entries, emptyValueIsDelete = false) { + for (let [k, v] of entries) { + k = k == null ? '' : String(k); + v = v == null ? '' : String(v); + if (!v && emptyValueIsDelete) { + this.delete(k); + } else { + this.set(k, v); + } + } + return this; + } + + /** + * @param {AttributeString} str - The attribute string identifying the attributes to insert into + * this map. + * @param {boolean} [emptyValueIsDelete] - If true and an entry's value is the empty string, the + * key is removed from this map (if present). + * @returns {AttributeMap} `this` (for chaining). + */ + updateFromString(str, emptyValueIsDelete = false) { + return this.update(attributes.attribsFromString(str, this.pool), emptyValueIsDelete); + } +} + +module.exports = AttributeMap; diff --git a/src/static/js/attributes.js b/src/static/js/attributes.js new file mode 100644 index 000000000..4ab347019 --- /dev/null +++ b/src/static/js/attributes.js @@ -0,0 +1,130 @@ +'use strict'; + +// Low-level utilities for manipulating attribute strings. For a high-level API, see AttributeMap. + +/** + * A `[key, value]` pair of strings describing a text attribute. + * + * @typedef {[string, string]} Attribute + */ + +/** + * A concatenated sequence of zero or more attribute identifiers, each one represented by an + * asterisk followed by a base-36 encoded attribute number. + * + * Examples: '', '*0', '*3*j*z*1q' + * + * @typedef {string} AttributeString + */ + +/** + * Converts an attribute string into a sequence of attribute identifier numbers. + * + * WARNING: This only works on attribute strings. It does NOT work on serialized operations or + * changesets. + * + * @param {AttributeString} str - Attribute string. + * @yields {number} The attribute numbers (to look up in the associated pool), in the order they + * appear in `str`. + * @returns {Generator} + */ +exports.decodeAttribString = function* (str) { + const re = /\*([0-9a-z]+)|./gy; + let match; + while ((match = re.exec(str)) != null) { + const [m, n] = match; + if (n == null) throw new Error(`invalid character in attribute string: ${m}`); + yield Number.parseInt(n, 36); + } +}; + +const checkAttribNum = (n) => { + if (typeof n !== 'number') throw new TypeError(`not a number: ${n}`); + if (n < 0) throw new Error(`attribute number is negative: ${n}`); + if (n !== Math.trunc(n)) throw new Error(`attribute number is not an integer: ${n}`); +}; + +/** + * Inverse of `decodeAttribString`. + * + * @param {Iterable} attribNums - Sequence of attribute numbers. + * @returns {AttributeString} + */ +exports.encodeAttribString = (attribNums) => { + let str = ''; + for (const n of attribNums) { + checkAttribNum(n); + str += `*${n.toString(36).toLowerCase()}`; + } + return str; +}; + +/** + * Converts a sequence of attribute numbers into a sequence of attributes. + * + * @param {Iterable} attribNums - Attribute numbers to look up in the pool. + * @param {AttributePool} pool - Attribute pool. + * @yields {Attribute} The identified attributes, in the same order as `attribNums`. + * @returns {Generator} + */ +exports.attribsFromNums = function* (attribNums, pool) { + for (const n of attribNums) { + checkAttribNum(n); + const attrib = pool.getAttrib(n); + if (attrib == null) throw new Error(`attribute ${n} does not exist in pool`); + yield attrib; + } +}; + +/** + * Inverse of `attribsFromNums`. + * + * @param {Iterable} attribs - Attributes. Any attributes not already in `pool` are + * inserted into `pool`. No checking is performed to ensure that the attributes are in the + * canonical order and that there are no duplicate keys. (Use an AttributeMap and/or `sort()` if + * required.) + * @param {AttributePool} pool - Attribute pool. + * @yields {number} The attribute number of each attribute in `attribs`, in order. + * @returns {Generator} + */ +exports.attribsToNums = function* (attribs, pool) { + for (const attrib of attribs) yield pool.putAttrib(attrib); +}; + +/** + * Convenience function that is equivalent to `attribsFromNums(decodeAttribString(str), pool)`. + * + * WARNING: This only works on attribute strings. It does NOT work on serialized operations or + * changesets. + * + * @param {AttributeString} str - Attribute string. + * @param {AttributePool} pool - Attribute pool. + * @yields {Attribute} The attributes identified in `str`, in order. + * @returns {Generator} + */ +exports.attribsFromString = function* (str, pool) { + yield* exports.attribsFromNums(exports.decodeAttribString(str), pool); +}; + +/** + * Inverse of `attribsFromString`. + * + * @param {Iterable} attribs - Attributes. The attributes to insert into the pool (if + * necessary) and encode. No checking is performed to ensure that the attributes are in the + * canonical order and that there are no duplicate keys. (Use an AttributeMap and/or `sort()` if + * required.) + * @param {AttributePool} pool - Attribute pool. + * @returns {AttributeString} + */ +exports.attribsToString = + (attribs, pool) => exports.encodeAttribString(exports.attribsToNums(attribs, pool)); + +/** + * Sorts the attributes in canonical order. The order of entries with the same attribute name is + * unspecified. + * + * @param {Attribute[]} attribs - Attributes to sort in place. + * @returns {Attribute[]} `attribs` (for chaining). + */ +exports.sort = + (attribs) => attribs.sort(([keyA], [keyB]) => (keyA > keyB ? 1 : 0) - (keyA < keyB ? 1 : 0)); diff --git a/src/tests/frontend/specs/AttributeMap.js b/src/tests/frontend/specs/AttributeMap.js new file mode 100644 index 000000000..92ca68334 --- /dev/null +++ b/src/tests/frontend/specs/AttributeMap.js @@ -0,0 +1,178 @@ +'use strict'; + +const AttributeMap = require('../../../static/js/AttributeMap'); +const AttributePool = require('../../../static/js/AttributePool'); +const attributes = require('../../../static/js/attributes'); + +describe('AttributeMap', function () { + const attribs = [ + ['foo', 'bar'], + ['baz', 'bif'], + ['emptyValue', ''], + ]; + let pool; + + const getPoolSize = () => { + let n = 0; + pool.eachAttrib(() => ++n); + return n; + }; + + beforeEach(async function () { + pool = new AttributePool(); + for (let i = 0; i < attribs.length; ++i) expect(pool.putAttrib(attribs[i])).to.equal(i); + }); + + it('fromString works', async function () { + const got = AttributeMap.fromString('*0*1*2', pool); + for (const [k, v] of attribs) expect(got.get(k)).to.equal(v); + // Maps iterate in insertion order, so [...got] should be in the same order as attribs. + expect(JSON.stringify([...got])).to.equal(JSON.stringify(attribs)); + }); + + describe('set', function () { + it('stores the value', async function () { + const m = new AttributeMap(pool); + expect(m.size).to.equal(0); + m.set('k', 'v'); + expect(m.size).to.equal(1); + expect(m.get('k')).to.equal('v'); + }); + + it('reuses attributes in the pool', async function () { + expect(getPoolSize()).to.equal(attribs.length); + const m = new AttributeMap(pool); + const [k0, v0] = attribs[0]; + m.set(k0, v0); + expect(getPoolSize()).to.equal(attribs.length); + expect(m.size).to.equal(1); + expect(m.toString()).to.equal('*0'); + }); + + it('inserts new attributes into the pool', async function () { + const m = new AttributeMap(pool); + expect(getPoolSize()).to.equal(attribs.length); + m.set('k', 'v'); + expect(getPoolSize()).to.equal(attribs.length + 1); + expect(JSON.stringify(pool.getAttrib(attribs.length))).to.equal(JSON.stringify(['k', 'v'])); + }); + + describe('coerces key and value to string', function () { + const testCases = [ + ['object (with toString)', {toString: () => 'obj'}, 'obj'], + ['undefined', undefined, ''], + ['null', null, ''], + ['boolean', true, 'true'], + ['number', 1, '1'], + ]; + for (const [desc, input, want] of testCases) { + describe(desc, function () { + it('key is coerced to string', async function () { + const m = new AttributeMap(pool); + m.set(input, 'value'); + expect(m.get(want)).to.equal('value'); + }); + + it('value is coerced to string', async function () { + const m = new AttributeMap(pool); + m.set('key', input); + expect(m.get('key')).to.equal(want); + }); + }); + } + }); + + it('returns the map', async function () { + const m = new AttributeMap(pool); + expect(m.set('k', 'v')).to.equal(m); + }); + }); + + describe('toString', function () { + it('sorts attributes', async function () { + const m = new AttributeMap(pool).update(attribs); + const got = [...attributes.attribsFromString(m.toString(), pool)]; + const want = attributes.sort([...attribs]); + // Verify that attribs is not already sorted so that this test doesn't accidentally pass. + expect(JSON.stringify(want)).to.not.equal(JSON.stringify(attribs)); + expect(JSON.stringify(got)).to.equal(JSON.stringify(want)); + }); + + it('returns all entries', async function () { + const m = new AttributeMap(pool); + expect(m.toString()).to.equal(''); + m.set(...attribs[0]); + expect(m.toString()).to.equal('*0'); + m.delete(attribs[0][0]); + expect(m.toString()).to.equal(''); + m.set(...attribs[1]); + expect(m.toString()).to.equal('*1'); + m.set(attribs[1][0], 'new value'); + expect(m.toString()).to.equal(attributes.encodeAttribString([attribs.length])); + m.set(...attribs[2]); + expect(m.toString()).to.equal(attributes.attribsToString( + attributes.sort([attribs[2], [attribs[1][0], 'new value']]), pool)); + }); + }); + + for (const funcName of ['update', 'updateFromString']) { + const callUpdateFn = (m, ...args) => { + if (funcName === 'updateFromString') { + args[0] = attributes.attribsToString(attributes.sort([...args[0]]), pool); + } + return AttributeMap.prototype[funcName].call(m, ...args); + }; + + describe(funcName, function () { + it('works', async function () { + const m = new AttributeMap(pool); + m.set(attribs[2][0], 'value to be overwritten'); + callUpdateFn(m, attribs); + for (const [k, v] of attribs) expect(m.get(k)).to.equal(v); + expect(m.size).to.equal(attribs.length); + const wantStr = attributes.attribsToString(attributes.sort([...attribs]), pool); + expect(m.toString()).to.equal(wantStr); + callUpdateFn(m, []); + expect(m.toString()).to.equal(wantStr); + }); + + it('inserts new attributes into the pool', async function () { + const m = new AttributeMap(pool); + callUpdateFn(m, [['k', 'v']]); + expect(m.size).to.equal(1); + expect(m.get('k')).to.equal('v'); + expect(getPoolSize()).to.equal(attribs.length + 1); + expect(m.toString()).to.equal(attributes.encodeAttribString([attribs.length])); + }); + + it('returns the map', async function () { + const m = new AttributeMap(pool); + expect(callUpdateFn(m, [])).to.equal(m); + }); + + describe('emptyValueIsDelete=false inserts empty values', function () { + for (const emptyVal of ['', null, undefined]) { + it(emptyVal == null ? String(emptyVal) : JSON.stringify(emptyVal), async function () { + const m = new AttributeMap(pool); + m.set('k', 'v'); + callUpdateFn(m, [['k', emptyVal]]); + expect(m.size).to.equal(1); + expect(m.toString()).to.equal(attributes.attribsToString([['k', '']], pool)); + }); + } + }); + + describe('emptyValueIsDelete=true deletes entries', function () { + for (const emptyVal of ['', null, undefined]) { + it(emptyVal == null ? String(emptyVal) : JSON.stringify(emptyVal), async function () { + const m = new AttributeMap(pool); + m.set('k', 'v'); + callUpdateFn(m, [['k', emptyVal]], true); + expect(m.size).to.equal(0); + expect(m.toString()).to.equal(''); + }); + } + }); + }); + } +}); diff --git a/src/tests/frontend/specs/attributes.js b/src/tests/frontend/specs/attributes.js new file mode 100644 index 000000000..13058dbe3 --- /dev/null +++ b/src/tests/frontend/specs/attributes.js @@ -0,0 +1,343 @@ +'use strict'; + +const AttributePool = require('../../../static/js/AttributePool'); +const attributes = require('../../../static/js/attributes'); + +describe('attributes', function () { + const attribs = [['foo', 'bar'], ['baz', 'bif']]; + let pool; + + beforeEach(async function () { + pool = new AttributePool(); + for (let i = 0; i < attribs.length; ++i) expect(pool.putAttrib(attribs[i])).to.equal(i); + }); + + describe('decodeAttribString', function () { + it('is a generator function', async function () { + expect(attributes.decodeAttribString).to.be.a((function* () {}).constructor); + }); + + describe('rejects invalid attribute strings', function () { + const testCases = ['x', '*0+1', '*A', '*0$', '*', '0', '*-1']; + for (const tc of testCases) { + it(JSON.stringify(tc), async function () { + expect(() => [...attributes.decodeAttribString(tc)]) + .to.throwException(/invalid character/); + }); + } + }); + + describe('accepts valid attribute strings', function () { + const testCases = [ + ['', []], + ['*0', [0]], + ['*a', [10]], + ['*z', [35]], + ['*10', [36]], + [ + '*0*1*2*3*4*5*6*7*8*9*a*b*c*d*e*f*g*h*i*j*k*l*m*n*o*p*q*r*s*t*u*v*w*x*y*z*10', + [...Array(37).keys()], + ], + ]; + for (const [input, want] of testCases) { + it(`${JSON.stringify(input)} -> ${JSON.stringify(want)}`, async function () { + const got = [...attributes.decodeAttribString(input)]; + expect(JSON.stringify(got)).to.equal(JSON.stringify(want)); + }); + } + }); + }); + + describe('encodeAttribString', function () { + describe('accepts any kind of iterable', function () { + const testCases = [ + ['generator', (function* () { yield 0; yield 1; })()], + ['list', [0, 1]], + ['set', new Set([0, 1])], + ]; + for (const [desc, input] of testCases) { + it(desc, async function () { + expect(attributes.encodeAttribString(input)).to.equal('*0*1'); + }); + } + }); + + describe('rejects invalid inputs', function () { + const testCases = [ + [null, /.*/], // Different browsers may have different error messages. + [[-1], /is negative/], + [['0'], /not a number/], + [[null], /not a number/], + [[0.5], /not an integer/], + [[{}], /not a number/], + [[true], /not a number/], + ]; + for (const [input, wantErr] of testCases) { + it(JSON.stringify(input), async function () { + expect(() => attributes.encodeAttribString(input)).to.throwException(wantErr); + }); + } + }); + + describe('accepts valid inputs', function () { + const testCases = [ + [[], ''], + [[0], '*0'], + [[10], '*a'], + [[35], '*z'], + [[36], '*10'], + [ + [...Array(37).keys()], + '*0*1*2*3*4*5*6*7*8*9*a*b*c*d*e*f*g*h*i*j*k*l*m*n*o*p*q*r*s*t*u*v*w*x*y*z*10', + ], + ]; + for (const [input, want] of testCases) { + it(`${JSON.stringify(input)} -> ${JSON.stringify(want)}`, async function () { + expect(attributes.encodeAttribString(input)).to.equal(want); + }); + } + }); + }); + + describe('attribsFromNums', function () { + it('is a generator function', async function () { + expect(attributes.attribsFromNums).to.be.a((function* () {}).constructor); + }); + + describe('accepts any kind of iterable', function () { + const testCases = [ + ['generator', (function* () { yield 0; yield 1; })()], + ['list', [0, 1]], + ['set', new Set([0, 1])], + ]; + + for (const [desc, input] of testCases) { + it(desc, async function () { + const gotAttribs = [...attributes.attribsFromNums(input, pool)]; + expect(JSON.stringify(gotAttribs)).to.equal(JSON.stringify(attribs)); + }); + } + }); + + describe('rejects invalid inputs', function () { + const testCases = [ + [null, /.*/], // Different browsers may have different error messages. + [[-1], /is negative/], + [['0'], /not a number/], + [[null], /not a number/], + [[0.5], /not an integer/], + [[{}], /not a number/], + [[true], /not a number/], + [[9999], /does not exist in pool/], + ]; + for (const [input, wantErr] of testCases) { + it(JSON.stringify(input), async function () { + expect(() => [...attributes.attribsFromNums(input, pool)]).to.throwException(wantErr); + }); + } + }); + + describe('accepts valid inputs', function () { + const testCases = [ + [[], []], + [[0], [attribs[0]]], + [[1], [attribs[1]]], + [[0, 1], [attribs[0], attribs[1]]], + [[1, 0], [attribs[1], attribs[0]]], + ]; + for (const [input, want] of testCases) { + it(`${JSON.stringify(input)} -> ${JSON.stringify(want)}`, async function () { + const gotAttribs = [...attributes.attribsFromNums(input, pool)]; + expect(JSON.stringify(gotAttribs)).to.equal(JSON.stringify(want)); + }); + } + }); + }); + + describe('attribsToNums', function () { + it('is a generator function', async function () { + expect(attributes.attribsToNums).to.be.a((function* () {}).constructor); + }); + + describe('accepts any kind of iterable', function () { + const testCases = [ + ['generator', (function* () { yield attribs[0]; yield attribs[1]; })()], + ['list', [attribs[0], attribs[1]]], + ['set', new Set([attribs[0], attribs[1]])], + ]; + + for (const [desc, input] of testCases) { + it(desc, async function () { + const gotNums = [...attributes.attribsToNums(input, pool)]; + expect(JSON.stringify(gotNums)).to.equal(JSON.stringify([0, 1])); + }); + } + }); + + describe('rejects invalid inputs', function () { + const testCases = [null, [null]]; + for (const input of testCases) { + it(JSON.stringify(input), async function () { + expect(() => [...attributes.attribsToNums(input, pool)]).to.throwException(); + }); + } + }); + + describe('reuses existing pool entries', function () { + const testCases = [ + [[], []], + [[attribs[0]], [0]], + [[attribs[1]], [1]], + [[attribs[0], attribs[1]], [0, 1]], + [[attribs[1], attribs[0]], [1, 0]], + ]; + for (const [input, want] of testCases) { + it(`${JSON.stringify(input)} -> ${JSON.stringify(want)}`, async function () { + const got = [...attributes.attribsToNums(input, pool)]; + expect(JSON.stringify(got)).to.equal(JSON.stringify(want)); + }); + } + }); + + describe('inserts new attributes into the pool', function () { + const testCases = [ + [[['k', 'v']], [attribs.length]], + [[attribs[0], ['k', 'v']], [0, attribs.length]], + [[['k', 'v'], attribs[0]], [attribs.length, 0]], + ]; + for (const [input, want] of testCases) { + it(`${JSON.stringify(input)} -> ${JSON.stringify(want)}`, async function () { + const got = [...attributes.attribsToNums(input, pool)]; + expect(JSON.stringify(got)).to.equal(JSON.stringify(want)); + expect(JSON.stringify(pool.getAttrib(attribs.length))) + .to.equal(JSON.stringify(['k', 'v'])); + }); + } + }); + + describe('coerces key and value to string', function () { + const testCases = [ + ['object (with toString)', {toString: () => 'obj'}, 'obj'], + ['undefined', undefined, ''], + ['null', null, ''], + ['boolean', true, 'true'], + ['number', 1, '1'], + ]; + for (const [desc, inputVal, wantVal] of testCases) { + describe(desc, function () { + for (const [desc, inputAttribs, wantAttribs] of [ + ['key is coerced to string', [[inputVal, 'value']], [[wantVal, 'value']]], + ['value is coerced to string', [['key', inputVal]], [['key', wantVal]]], + ]) { + it(desc, async function () { + const gotNums = [...attributes.attribsToNums(inputAttribs, pool)]; + // Each attrib in inputAttribs is expected to be new to the pool. + const wantNums = [...Array(attribs.length + 1).keys()].slice(attribs.length); + expect(JSON.stringify(gotNums)).to.equal(JSON.stringify(wantNums)); + const gotAttribs = gotNums.map((n) => pool.getAttrib(n)); + expect(JSON.stringify(gotAttribs)).to.equal(JSON.stringify(wantAttribs)); + }); + } + }); + } + }); + }); + + describe('attribsFromString', function () { + it('is a generator function', async function () { + expect(attributes.attribsFromString).to.be.a((function* () {}).constructor); + }); + + describe('rejects invalid attribute strings', function () { + const testCases = [ + ['x', /invalid character/], + ['*0+1', /invalid character/], + ['*A', /invalid character/], + ['*0$', /invalid character/], + ['*', /invalid character/], + ['0', /invalid character/], + ['*-1', /invalid character/], + ['*9999', /does not exist in pool/], + ]; + for (const [input, wantErr] of testCases) { + it(JSON.stringify(input), async function () { + expect(() => [...attributes.attribsFromString(input, pool)]).to.throwException(wantErr); + }); + } + }); + + describe('accepts valid inputs', function () { + const testCases = [ + ['', []], + ['*0', [attribs[0]]], + ['*1', [attribs[1]]], + ['*0*1', [attribs[0], attribs[1]]], + ['*1*0', [attribs[1], attribs[0]]], + ]; + for (const [input, want] of testCases) { + it(`${JSON.stringify(input)} -> ${JSON.stringify(want)}`, async function () { + const gotAttribs = [...attributes.attribsFromString(input, pool)]; + expect(JSON.stringify(gotAttribs)).to.equal(JSON.stringify(want)); + }); + } + }); + }); + + describe('attribsToString', function () { + describe('accepts any kind of iterable', function () { + const testCases = [ + ['generator', (function* () { yield attribs[0]; yield attribs[1]; })()], + ['list', [attribs[0], attribs[1]]], + ['set', new Set([attribs[0], attribs[1]])], + ]; + + for (const [desc, input] of testCases) { + it(desc, async function () { + const got = attributes.attribsToString(input, pool); + expect(got).to.equal('*0*1'); + }); + } + }); + + describe('rejects invalid inputs', function () { + const testCases = [null, [null]]; + for (const input of testCases) { + it(JSON.stringify(input), async function () { + expect(() => attributes.attribsToString(input, pool)).to.throwException(); + }); + } + }); + + describe('reuses existing pool entries', function () { + const testCases = [ + [[], ''], + [[attribs[0]], '*0'], + [[attribs[1]], '*1'], + [[attribs[0], attribs[1]], '*0*1'], + [[attribs[1], attribs[0]], '*1*0'], + ]; + for (const [input, want] of testCases) { + it(`${JSON.stringify(input)} -> ${JSON.stringify(want)}`, async function () { + const got = attributes.attribsToString(input, pool); + expect(got).to.equal(want); + }); + } + }); + + describe('inserts new attributes into the pool', function () { + const testCases = [ + [[['k', 'v']], `*${attribs.length}`], + [[attribs[0], ['k', 'v']], `*0*${attribs.length}`], + [[['k', 'v'], attribs[0]], `*${attribs.length}*0`], + ]; + for (const [input, want] of testCases) { + it(`${JSON.stringify(input)} -> ${JSON.stringify(want)}`, async function () { + const got = attributes.attribsToString(input, pool); + expect(got).to.equal(want); + expect(JSON.stringify(pool.getAttrib(attribs.length))) + .to.equal(JSON.stringify(['k', 'v'])); + }); + } + }); + }); +}); From 1f227200da1690e5bcb6389a5f83be8bc2556171 Mon Sep 17 00:00:00 2001 From: Richard Hansen Date: Thu, 18 Nov 2021 20:06:56 -0500 Subject: [PATCH 014/446] Replace separate attrib key, value calls with single pair call --- src/node/utils/padDiff.js | 30 ++++++++---------------------- src/static/js/AttributeManager.js | 12 +++--------- src/static/js/linestylefilter.js | 7 ++----- 3 files changed, 13 insertions(+), 36 deletions(-) diff --git a/src/node/utils/padDiff.js b/src/node/utils/padDiff.js index 670e8d6a1..02fbdd99f 100644 --- a/src/node/utils/padDiff.js +++ b/src/node/utils/padDiff.js @@ -364,9 +364,6 @@ PadDiff.prototype._createDeletionChangeset = function (cs, startAText, apool) { }; }; - const attribKeys = []; - const attribValues = []; - // iterate over all operators of this changeset while (csIter.hasNext()) { const csOp = csIter.next(); @@ -381,28 +378,17 @@ PadDiff.prototype._createDeletionChangeset = function (cs, startAText, apool) { if (csOp.attribs && textBank !== '*') { const deletedAttrib = apool.putAttrib(['removed', true]); let authorAttrib = apool.putAttrib(['author', '']); - - attribKeys.length = 0; - attribValues.length = 0; + const attribs = []; Changeset.eachAttribNumber(csOp.attribs, (n) => { - attribKeys.push(apool.getAttribKey(n)); - attribValues.push(apool.getAttribValue(n)); - - if (apool.getAttribKey(n) === 'author') { - authorAttrib = n; - } + const attrib = apool.getAttrib(n); + attribs.push(attrib); + if (attrib[0] === 'author') authorAttrib = n; }); - - const undoBackToAttribs = cachedStrFunc((attribs) => { + const undoBackToAttribs = cachedStrFunc((oldAttribsStr) => { const backAttribs = []; - for (let i = 0; i < attribKeys.length; i++) { - const appliedKey = attribKeys[i]; - const appliedValue = attribValues[i]; - const oldValue = Changeset.attribsAttributeValue(attribs, appliedKey, apool); - - if (appliedValue !== oldValue) { - backAttribs.push([appliedKey, oldValue]); - } + for (const [key, value] of attribs) { + const oldValue = Changeset.attribsAttributeValue(oldAttribsStr, key, apool); + if (oldValue !== value) backAttribs.push([key, oldValue]) } return Changeset.makeAttribsString('=', backAttribs, apool); diff --git a/src/static/js/AttributeManager.js b/src/static/js/AttributeManager.js index d5b650bd5..760422427 100644 --- a/src/static/js/AttributeManager.js +++ b/src/static/js/AttributeManager.js @@ -166,9 +166,7 @@ AttributeManager.prototype = _(AttributeManager.prototype).extend({ const op = opIter.next(); if (!op.attribs) return []; const attributes = []; - Changeset.eachAttribNumber(op.attribs, (n) => { - attributes.push([this.rep.apool.getAttribKey(n), this.rep.apool.getAttribValue(n)]); - }); + Changeset.eachAttribNumber(op.attribs, (n) => attributes.push(this.rep.apool.getAttrib(n))); return attributes; }, @@ -277,12 +275,8 @@ AttributeManager.prototype = _(AttributeManager.prototype).extend({ currentPointer += currentOperation.chars; if (currentPointer <= column) continue; const attributes = []; - Changeset.eachAttribNumber(currentOperation.attribs, (n) => { - attributes.push([ - this.rep.apool.getAttribKey(n), - this.rep.apool.getAttribValue(n), - ]); - }); + Changeset.eachAttribNumber( + currentOperation.attribs, (n) => attributes.push(this.rep.apool.getAttrib(n))); return attributes; } return []; diff --git a/src/static/js/linestylefilter.js b/src/static/js/linestylefilter.js index c0a81e9da..0821889c7 100644 --- a/src/static/js/linestylefilter.js +++ b/src/static/js/linestylefilter.js @@ -75,11 +75,8 @@ linestylefilter.getLineStyleFilter = (lineLength, aline, textAndClassFunc, apool // For each attribute number Changeset.eachAttribNumber(attribs, (n) => { - // Give us this attributes key - const key = apool.getAttribKey(n); - if (!key) return; - const value = apool.getAttribValue(n); - if (!value) return; + const [key, value] = apool.getAttrib(n); + if (!key || !value) return; if (!isLineAttribMarker && AttributeManager.lineAttributes.indexOf(key) >= 0) { isLineAttribMarker = true; } From f40d2851099421bdc491c82a4ccac12d467e0e0f Mon Sep 17 00:00:00 2001 From: Richard Hansen Date: Fri, 19 Nov 2021 05:16:11 -0500 Subject: [PATCH 015/446] tests: Refine contentcollector tests --- src/tests/backend/specs/contentcollector.js | 307 ++++++++++++-------- 1 file changed, 189 insertions(+), 118 deletions(-) diff --git a/src/tests/backend/specs/contentcollector.js b/src/tests/backend/specs/contentcollector.js index f7bc539e6..a6ff8c2d7 100644 --- a/src/tests/backend/specs/contentcollector.js +++ b/src/tests/backend/specs/contentcollector.js @@ -10,35 +10,64 @@ */ const AttributePool = require('../../../static/js/AttributePool'); +const Changeset = require('../../../static/js/Changeset'); const assert = require('assert').strict; const contentcollector = require('../../../static/js/contentcollector'); const jsdom = require('jsdom'); -const tests = { - nestedLi: { +// All test case `wantAlines` values must only refer to attributes in this list so that the +// attribute numbers do not change due to changes in pool insertion order. +const knownAttribs = [ + ['insertorder', 'first'], + ['italic', 'true'], + ['list', 'bullet1'], + ['list', 'bullet2'], + ['list', 'number1'], + ['list', 'number2'], + ['lmkr', '1'], + ['start', '1'], + ['start', '2'], +]; + +const testCases = [ + { + description: 'Simple', + html: '

foo

', + wantAlines: ['+3'], + wantText: ['foo'], + }, + { + description: 'Line starts with asterisk', + html: '

*foo

', + wantAlines: ['+4'], + wantText: ['*foo'], + }, + { description: 'Complex nested Li', html: '
  1. one
    1. 1.1
  3. two
', - wantLineAttribs: [ - '*0*1*2*3+1+3', '*0*4*2*5+1+3', '*0*1*2*5+1+3', + wantAlines: [ + '*0*4*6*7+1+3', + '*0*5*6*8+1+3', + '*0*4*6*8+1+3', ], wantText: [ '*one', '*1.1', '*two', ], }, - complexNest: { + { description: 'Complex list of different types', html: '
  • one
  • two
  • 0
  • 1
  • 2
    • 3
    • 4
  1. item
    1. item1
    2. item2
', - wantLineAttribs: [ - '*0*1*2+1+3', - '*0*1*2+1+3', - '*0*1*2+1+1', - '*0*1*2+1+1', - '*0*1*2+1+1', - '*0*3*2+1+1', - '*0*3*2+1+1', - '*0*4*2*5+1+4', - '*0*6*2*7+1+5', - '*0*6*2*7+1+5', + wantAlines: [ + '*0*2*6+1+3', + '*0*2*6+1+3', + '*0*2*6+1+1', + '*0*2*6+1+1', + '*0*2*6+1+1', + '*0*3*6+1+1', + '*0*3*6+1+1', + '*0*4*6*7+1+4', + '*0*5*6*8+1+5', + '*0*5*6*8+1+5', ], wantText: [ '*one', @@ -53,147 +82,174 @@ const tests = { '*item2', ], }, - ul: { + { description: 'Tests if uls properly get attributes', html: '
  • a
  • b
div

foo

', - wantLineAttribs: ['*0*1*2+1+1', '*0*1*2+1+1', '+3', '+3'], + wantAlines: [ + '*0*2*6+1+1', + '*0*2*6+1+1', + '+3', + '+3', + ], wantText: ['*a', '*b', 'div', 'foo'], }, - ulIndented: { + { description: 'Tests if indented uls properly get attributes', html: '
  • a
    • b
  • a

foo

', - wantLineAttribs: ['*0*1*2+1+1', '*0*3*2+1+1', '*0*1*2+1+1', '+3'], + wantAlines: [ + '*0*2*6+1+1', + '*0*3*6+1+1', + '*0*2*6+1+1', + '+3', + ], wantText: ['*a', '*b', '*a', 'foo'], }, - ol: { + { description: 'Tests if ols properly get line numbers when in a normal OL', html: '
  1. a
  2. b
  3. c

test

', - wantLineAttribs: ['*0*1*2*3+1+1', '*0*1*2*3+1+1', '*0*1*2*3+1+1', '+4'], + wantAlines: [ + '*0*4*6*7+1+1', + '*0*4*6*7+1+1', + '*0*4*6*7+1+1', + '+4', + ], wantText: ['*a', '*b', '*c', 'test'], noteToSelf: 'Ensure empty P does not induce line attribute marker, wont this break the editor?', }, - lineDoBreakInOl: { + { description: 'A single completely empty line break within an ol should reset count if OL is closed off..', html: '
  1. should be 1

hello

  1. should be 1
  2. should be 2

', - wantLineAttribs: ['*0*1*2*3+1+b', '+5', '*0*1*2*4+1+b', '*0*1*2*4+1+b', ''], + wantAlines: [ + '*0*4*6*7+1+b', + '+5', + '*0*4*6*8+1+b', + '*0*4*6*8+1+b', + '', + ], wantText: ['*should be 1', 'hello', '*should be 1', '*should be 2', ''], noteToSelf: "Shouldn't include attribute marker in the

line", }, - testP: { + { description: 'A single

should create a new line', html: '

', - wantLineAttribs: ['', ''], + wantAlines: ['', ''], wantText: ['', ''], noteToSelf: '

should create a line break but not break numbering', }, - nestedOl: { - description: 'Tests if ols properly get line numbers when in a normal OL', + { + description: 'Tests if ols properly get line numbers when in a normal OL #2', html: 'a
  1. b
    1. c
notlist

foo

', - wantLineAttribs: ['+1', '*0*1*2*3+1+1', '*0*4*2*5+1+1', '+7', '+3'], + wantAlines: [ + '+1', + '*0*4*6*7+1+1', + '*0*5*6*8+1+1', + '+7', + '+3', + ], wantText: ['a', '*b', '*c', 'notlist', 'foo'], noteToSelf: 'Ensure empty P does not induce line attribute marker, wont this break the editor?', }, - nestedOl2: { + { description: 'First item being an UL then subsequent being OL will fail', html: '
  • a
    1. b
    2. c
', - wantLineAttribs: ['+1', '*0*1*2*3+1+1', '*0*4*2*5+1+1'], + wantAlines: ['+1', '*0*1*2*3+1+1', '*0*4*2*5+1+1'], wantText: ['a', '*b', '*c'], noteToSelf: 'Ensure empty P does not induce line attribute marker, wont this break the editor?', disabled: true, }, - lineDontBreakOL: { + { description: 'A single completely empty line break within an ol should NOT reset count', html: '
  1. should be 1

  2. should be 2
  3. should be 3

', - wantLineAttribs: [], + wantAlines: [], wantText: ['*should be 1', '*should be 2', '*should be 3'], noteToSelf: "

should create a line break but not break numbering -- This is what I can't get working!", disabled: true, }, - ignoreAnyTagsOutsideBody: { + { description: 'Content outside body should be ignored', html: 'titleempty
', - wantLineAttribs: ['+5'], + wantAlines: ['+5'], wantText: ['empty'], }, - lineWithMultipleSpaces: { + { description: 'Multiple spaces should be preserved', html: 'Text with more than one space.
', - wantLineAttribs: ['+10'], + wantAlines: ['+10'], wantText: ['Text with more than one space.'], }, - lineWithMultipleNonBreakingAndNormalSpaces: { + { description: 'non-breaking and normal space should be preserved', html: 'Text with  more   than  one space.
', - wantLineAttribs: ['+10'], + wantAlines: ['+10'], wantText: ['Text with more than one space.'], }, - multiplenbsp: { + { description: 'Multiple nbsp should be preserved', html: '  
', - wantLineAttribs: ['+2'], + wantAlines: ['+2'], wantText: [' '], }, - multipleNonBreakingSpaceBetweenWords: { + { description: 'Multiple nbsp between words ', html: '  word1  word2   word3
', - wantLineAttribs: ['+m'], + wantAlines: ['+m'], wantText: [' word1 word2 word3'], }, - nonBreakingSpacePreceededBySpaceBetweenWords: { + { description: 'A non-breaking space preceded by a normal space', html: '  word1  word2  word3
', - wantLineAttribs: ['+l'], + wantAlines: ['+l'], wantText: [' word1 word2 word3'], }, - nonBreakingSpaceFollowededBySpaceBetweenWords: { + { description: 'A non-breaking space followed by a normal space', html: '  word1  word2  word3
', - wantLineAttribs: ['+l'], + wantAlines: ['+l'], wantText: [' word1 word2 word3'], }, - spacesAfterNewline: { + { description: 'Don\'t collapse spaces that follow a newline', html: 'something
something
', - wantLineAttribs: ['+9', '+m'], + wantAlines: ['+9', '+m'], wantText: ['something', ' something'], }, - spacesAfterNewlineP: { + { description: 'Don\'t collapse spaces that follow a empty paragraph', html: 'something

something
', - wantLineAttribs: ['+9', '', '+m'], + wantAlines: ['+9', '', '+m'], wantText: ['something', '', ' something'], }, - spacesAtEndOfLine: { + { description: 'Don\'t collapse spaces that preceed/follow a newline', html: 'something
something
', - wantLineAttribs: ['+l', '+m'], + wantAlines: ['+l', '+m'], wantText: ['something ', ' something'], }, - spacesAtEndOfLineP: { + { description: 'Don\'t collapse spaces that preceed/follow a empty paragraph', html: 'something

something
', - wantLineAttribs: ['+l', '', '+m'], + wantAlines: ['+l', '', '+m'], wantText: ['something ', '', ' something'], }, - nonBreakingSpacesAfterNewlines: { + { description: 'Don\'t collapse non-breaking spaces that follow a newline', html: 'something
   something
', - wantLineAttribs: ['+9', '+c'], + wantAlines: ['+9', '+c'], wantText: ['something', ' something'], }, - nonBreakingSpacesAfterNewlinesP: { + { description: 'Don\'t collapse non-breaking spaces that follow a paragraph', html: 'something

   something
', - wantLineAttribs: ['+9', '', '+c'], + wantAlines: ['+9', '', '+c'], wantText: ['something', '', ' something'], }, - preserveSpacesInsideElements: { + { description: 'Preserve all spaces when multiple are present', html: 'Need more space s !
', - wantLineAttribs: ['+h*0+4+2'], + wantAlines: ['+h*1+4+2'], wantText: ['Need more space s !'], }, - preserveSpacesAcrossNewlines: { + { description: 'Newlines and multiple spaces across newlines should be preserved', html: ` Need @@ -201,25 +257,25 @@ const tests = { space s !
`, - wantLineAttribs: ['+19*0+4+b'], + wantAlines: ['+19*1+4+b'], wantText: ['Need more space s !'], }, - multipleNewLinesAtBeginning: { + { description: 'Multiple new lines at the beginning should be preserved', html: '

first line

second line
', - wantLineAttribs: ['', '', '', '', '+a', '', '+b'], + wantAlines: ['', '', '', '', '+a', '', '+b'], wantText: ['', '', '', '', 'first line', '', 'second line'], }, - multiLineParagraph: { + { description: 'A paragraph with multiple lines should not loose spaces when lines are combined', html: `

а б в г ґ д е є ж з и і ї й к л м н о п р с т у ф х ц ч ш щ ю я ь

`, - wantLineAttribs: ['+1t'], + wantAlines: ['+1t'], wantText: ['а б в г ґ д е є ж з и і ї й к л м н о п р с т у ф х ц ч ш щ ю я ь'], }, - multiLineParagraphWithPre: { + { description: 'lines in preformatted text should be kept intact', html: `

а б в г ґ д е є ж з и і ї й к л м н о

multiple
@@ -229,7 +285,7 @@ pre

п р с т у ф х ц ч ш щ ю я ь

`, - wantLineAttribs: ['+11', '+8', '+5', '+2', '+3', '+r'], + wantAlines: ['+11', '+8', '+5', '+2', '+3', '+r'], wantText: [ 'а б в г ґ д е є ж з и і ї й к л м н о', 'multiple', @@ -239,85 +295,100 @@ pre 'п р с т у ф х ц ч ш щ ю я ь', ], }, - preIntroducesASpace: { + { description: 'pre should be on a new line not preceded by a space', html: `

1 

preline
`, - wantLineAttribs: ['+6', '+7'], + wantAlines: ['+6', '+7'], wantText: [' 1 ', 'preline'], }, - dontDeleteSpaceInsideElements: { + { description: 'Preserve spaces on the beginning and end of a element', html: 'Need more space s !
', - wantLineAttribs: ['+f*0+3+1'], + wantAlines: ['+f*1+3+1'], wantText: ['Need more space s !'], }, - dontDeleteSpaceOutsideElements: { + { description: 'Preserve spaces outside elements', html: 'Need more space s !
', - wantLineAttribs: ['+g*0+1+2'], + wantAlines: ['+g*1+1+2'], wantText: ['Need more space s !'], }, - dontDeleteSpaceAtEndOfElement: { + { description: 'Preserve spaces at the end of an element', html: 'Need more space s !
', - wantLineAttribs: ['+g*0+2+1'], + wantAlines: ['+g*1+2+1'], wantText: ['Need more space s !'], }, - dontDeleteSpaceAtBeginOfElements: { + { description: 'Preserve spaces at the start of an element', html: 'Need more space s !
', - wantLineAttribs: ['+f*0+2+2'], + wantAlines: ['+f*1+2+2'], wantText: ['Need more space s !'], }, -}; +]; describe(__filename, function () { - for (const test of Object.keys(tests)) { - const testObj = tests[test]; - describe(test, function () { - if (testObj.disabled) { - return xit('DISABLED:', test, function (done) { - done(); - }); - } + for (const tc of testCases) { + describe(tc.description, function () { + let apool; + let result; - it(testObj.description, async function () { - const {window: {document}} = new jsdom.JSDOM(testObj.html); - // Create an empty attribute pool - const apool = new AttributePool(); - // Convert a dom tree into a list of lines and attribute liens - // using the content collector object + before(async function () { + if (tc.disabled) return this.skip(); + const {window: {document}} = new jsdom.JSDOM(tc.html); + apool = new AttributePool(); + // To reduce test fragility, the attribute pool is seeded with `knownAttribs`, and all + // attributes in `tc.wantAlines` must be in `knownAttribs`. (This guarantees that attribute + // numbers do not change if the attribute processing code changes.) + for (const attrib of knownAttribs) apool.putAttrib(attrib); + for (const aline of tc.wantAlines) { + const opIter = Changeset.opIterator(aline); + while (opIter.hasNext()) { + const op = opIter.next(); + Changeset.eachAttribNumber(op.attribs, (n) => assert(n < knownAttribs.length)); + } + } const cc = contentcollector.makeContentCollector(true, null, apool); cc.collectContent(document.body); - const result = cc.finish(); - const gotAttributes = result.lineAttribs; - const wantAttributes = testObj.wantLineAttribs; - const gotText = new Array(result.lines); - const wantText = testObj.wantText; + result = cc.finish(); + }); - assert.deepEqual(gotText[0], wantText); - assert.deepEqual(gotAttributes, wantAttributes); + it('text matches', async function () { + assert.deepEqual(result.lines, tc.wantText); + }); + + it('alines match', async function () { + assert.deepEqual(result.lineAttribs, tc.wantAlines); + }); + + it('attributes are sorted in canonical order', async function () { + const gotAttribs = []; + const wantAttribs = []; + for (const aline of result.lineAttribs) { + const gotAlineAttribs = []; + gotAttribs.push(gotAlineAttribs); + const wantAlineAttribs = []; + wantAttribs.push(wantAlineAttribs); + const opIter = Changeset.opIterator(aline); + while (opIter.hasNext()) { + const op = opIter.next(); + const gotOpAttribs = []; + gotAlineAttribs.push(gotOpAttribs); + const wantOpAttribs = []; + wantAlineAttribs.push(wantOpAttribs); + Changeset.eachAttribNumber(op.attribs, (n) => { + const attrib = apool.getAttrib(n); + gotOpAttribs.push(attrib); + wantOpAttribs.push(attrib); + }); + wantOpAttribs.sort(([keyA], [keyB]) => (keyA > keyB ? 1 : 0) - (keyA < keyB ? 1 : 0)); + } + } + assert.deepEqual(gotAttribs, wantAttribs); }); }); } }); - - -function arraysEqual(a, b) { - if (a === b) return true; - if (a == null || b == null) return false; - if (a.length !== b.length) return false; - - // If you don't care about the order of the elements inside - // the array, you should sort both arrays here. - // Please note that calling sort on an array will modify that array. - // you might want to clone your array first. - - for (let i = 0; i < a.length; ++i) { - if (a[i] !== b[i]) return false; - } - return true; -} From f1eb7a25a6490941021a3c777e52fc00c4cfd480 Mon Sep 17 00:00:00 2001 From: Richard Hansen Date: Fri, 19 Nov 2021 00:51:25 -0500 Subject: [PATCH 016/446] Changeset: Migrate to the new attribute API --- CHANGELOG.md | 12 ++ doc/api/hooks_server-side.md | 3 +- src/node/handler/PadMessageHandler.js | 38 ++---- src/node/utils/ExportHelper.js | 6 +- src/node/utils/ExportHtml.js | 5 +- src/node/utils/ExportTxt.js | 5 +- src/node/utils/padDiff.js | 43 +++---- src/static/js/AttributeManager.js | 18 +-- src/static/js/Changeset.js | 131 ++++++++++---------- src/static/js/ace2_inner.js | 28 ++--- src/static/js/broadcast.js | 25 ++-- src/static/js/changesettracker.js | 20 ++- src/static/js/contentcollector.js | 26 ++-- src/static/js/linestylefilter.js | 9 +- src/tests/backend/specs/contentcollector.js | 16 +-- 15 files changed, 175 insertions(+), 210 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 8850e4036..799b875e3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,18 @@ (low-level API) and `ep_etherpad-lite/static/js/AttributeMap` (high-level API). +### Compatibility changes + +#### For plugin authors + +* Changes to the `src/static/js/Changeset.js` library: + * The following attribute processing functions are deprecated (use the new + attribute APIs instead): + * `attribsAttributeValue()` + * `eachAttribNumber()` + * `makeAttribsString()` + * `opAttributeValue()` + # 1.8.15 ### Security fixes diff --git a/doc/api/hooks_server-side.md b/doc/api/hooks_server-side.md index 476c3d050..38cca9baa 100644 --- a/doc/api/hooks_server-side.md +++ b/doc/api/hooks_server-side.md @@ -665,6 +665,7 @@ Context properties: Example: ```javascript +const AttributeMap = require('ep_etherpad-lite/static/js/AttributeMap'); const Changeset = require('ep_etherpad-lite/static/js/Changeset'); exports.getLineHTMLForExport = async (hookName, context) => { @@ -672,7 +673,7 @@ exports.getLineHTMLForExport = async (hookName, context) => { const opIter = Changeset.opIterator(context.attribLine); if (!opIter.hasNext()) return; const op = opIter.next(); - const heading = Changeset.opAttributeValue(op, 'heading', apool); + const heading = AttributeMap.fromString(op.attribs, context.apool).get('heading'); if (!heading) return; context.lineContent = `<${heading}>${context.lineContent}`; }; diff --git a/src/node/handler/PadMessageHandler.js b/src/node/handler/PadMessageHandler.js index 0735ce97f..868ca5b4a 100644 --- a/src/node/handler/PadMessageHandler.js +++ b/src/node/handler/PadMessageHandler.js @@ -19,6 +19,7 @@ * limitations under the License. */ +const AttributeMap = require('../../static/js/AttributeMap'); const padManager = require('../db/PadManager'); const Changeset = require('../../static/js/Changeset'); const ChatMessage = require('../../static/js/ChatMessage'); @@ -32,7 +33,6 @@ const plugins = require('../../static/js/pluginfw/plugin_defs.js'); const log4js = require('log4js'); const messageLogger = log4js.getLogger('message'); const accessLogger = log4js.getLogger('access'); -const _ = require('underscore'); const hooks = require('../../static/js/pluginfw/hooks.js'); const stats = require('../stats'); const assert = require('assert').strict; @@ -585,14 +585,6 @@ const handleUserChanges = async (socket, message) => { // Verify that the changeset has valid syntax and is in canonical form Changeset.checkRep(changeset); - // Verify that the attribute indexes used in the changeset are all - // defined in the accompanying attribute pool. - Changeset.eachAttribNumber(changeset, (n) => { - if (!wireApool.getAttrib(n)) { - throw new Error(`Attribute pool is missing attribute ${n} for changeset ${changeset}`); - } - }); - // Validate all added 'author' attribs to be the same value as the current user const iterator = Changeset.opIterator(Changeset.unpack(changeset).ops); let op; @@ -605,19 +597,14 @@ const handleUserChanges = async (socket, message) => { // - can have attribs, but they are discarded and don't show up in the attribs - // but do show up in the pool - op.attribs.split('*').forEach((attr) => { - if (!attr) return; - - attr = wireApool.getAttrib(Changeset.parseNum(attr)); - if (!attr) return; - - // the empty author is used in the clearAuthorship functionality so this - // should be the only exception - if ('author' === attr[0] && (attr[1] !== thisSession.author && attr[1] !== '')) { - throw new Error(`Author ${thisSession.author} tried to submit changes as author ` + - `${attr[1]} in changeset ${changeset}`); - } - }); + // Besides verifying the author attribute, this serves a second purpose: + // AttributeMap.fromString() ensures that all attribute numbers are valid (it will throw if + // an attribute number isn't in the pool). + const opAuthorId = AttributeMap.fromString(op.attribs, wireApool).get('author'); + if (opAuthorId && opAuthorId !== thisSession.author) { + throw new Error(`Author ${thisSession.author} tried to submit changes as author ` + + `${opAuthorId} in changeset ${changeset}`); + } } // ex. adoptChangesetAttribs @@ -758,11 +745,8 @@ const _correctMarkersInPad = (atext, apool) => { let offset = 0; while (iter.hasNext()) { const op = iter.next(); - - const hasMarker = _.find( - AttributeManager.lineAttributes, - (attribute) => Changeset.opAttributeValue(op, attribute, apool)) !== undefined; - + const attribs = AttributeMap.fromString(op.attribs, apool); + const hasMarker = AttributeManager.lineAttributes.some((a) => attribs.has(a)); if (hasMarker) { for (let i = 0; i < op.chars; i++) { if (offset > 0 && text.charAt(offset - 1) !== '\n') { diff --git a/src/node/utils/ExportHelper.js b/src/node/utils/ExportHelper.js index ba71269d1..401fad70b 100644 --- a/src/node/utils/ExportHelper.js +++ b/src/node/utils/ExportHelper.js @@ -19,6 +19,7 @@ * limitations under the License. */ +const AttributeMap = require('../../static/js/AttributeMap'); const Changeset = require('../../static/js/Changeset'); exports.getPadPlainText = (pad, revNum) => { @@ -54,7 +55,8 @@ exports._analyzeLine = (text, aline, apool) => { const opIter = Changeset.opIterator(aline); if (opIter.hasNext()) { const op = opIter.next(); - let listType = Changeset.opAttributeValue(op, 'list', apool); + const attribs = AttributeMap.fromString(op.attribs, apool); + let listType = attribs.get('list'); if (listType) { lineMarker = 1; listType = /([a-z]+)([0-9]+)/.exec(listType); @@ -63,7 +65,7 @@ exports._analyzeLine = (text, aline, apool) => { line.listLevel = Number(listType[2]); } } - const start = Changeset.opAttributeValue(op, 'start', apool); + const start = attribs.get('start'); if (start) { line.start = start; } diff --git a/src/node/utils/ExportHtml.js b/src/node/utils/ExportHtml.js index bc50da77b..800798f9c 100644 --- a/src/node/utils/ExportHtml.js +++ b/src/node/utils/ExportHtml.js @@ -16,6 +16,7 @@ */ const Changeset = require('../../static/js/Changeset'); +const attributes = require('../../static/js/attributes'); const padManager = require('../db/PadManager'); const _ = require('underscore'); const Security = require('../../static/js/security'); @@ -206,11 +207,11 @@ const getHTMLFromAtext = async (pad, atext, authorColors) => { const usedAttribs = []; // mark all attribs as used - Changeset.eachAttribNumber(o.attribs, (a) => { + for (const a of attributes.decodeAttribString(o.attribs)) { if (a in anumMap) { usedAttribs.push(anumMap[a]); // i = 0 => bold, etc. } - }); + } let outermostTag = -1; // find the outer most open tag that is no longer used for (let i = openTags.length - 1; i >= 0; i--) { diff --git a/src/node/utils/ExportTxt.js b/src/node/utils/ExportTxt.js index 0ff7ded83..1d7ce5469 100644 --- a/src/node/utils/ExportTxt.js +++ b/src/node/utils/ExportTxt.js @@ -20,6 +20,7 @@ */ const Changeset = require('../../static/js/Changeset'); +const attributes = require('../../static/js/attributes'); const padManager = require('../db/PadManager'); const _analyzeLine = require('./ExportHelper')._analyzeLine; @@ -82,7 +83,7 @@ const getTXTFromAtext = (pad, atext, authorColors) => { const o = iter.next(); let propChanged = false; - Changeset.eachAttribNumber(o.attribs, (a) => { + for (const a of attributes.decodeAttribString(o.attribs)) { if (a in anumMap) { const i = anumMap[a]; // i = 0 => bold, etc. @@ -93,7 +94,7 @@ const getTXTFromAtext = (pad, atext, authorColors) => { propVals[i] = STAY; } } - }); + } for (let i = 0; i < propVals.length; i++) { if (propVals[i] === true) { diff --git a/src/node/utils/padDiff.js b/src/node/utils/padDiff.js index 02fbdd99f..a8841cf22 100644 --- a/src/node/utils/padDiff.js +++ b/src/node/utils/padDiff.js @@ -1,5 +1,8 @@ 'use strict'; + +const AttributeMap = require('../../static/js/AttributeMap'); const Changeset = require('../../static/js/Changeset'); +const attributes = require('../../static/js/attributes'); const exportHtml = require('./ExportHtml'); function PadDiff(pad, fromRev, toRev) { @@ -54,17 +57,11 @@ PadDiff.prototype._isClearAuthorship = function (changeset) { return false; } - const attributes = []; - Changeset.eachAttribNumber(changeset, (attrNum) => { - attributes.push(attrNum); - }); + const [appliedAttribute, anotherAttribute] = + attributes.attribsFromString(clearOperator.attribs, this._pad.pool); - // check that this changeset uses only one attribute - if (attributes.length !== 1) { - return false; - } - - const appliedAttribute = this._pad.pool.getAttrib(attributes[0]); + // Check that the operation has exactly one attribute. + if (appliedAttribute == null || anotherAttribute != null) return false; // check if the applied attribute is an anonymous author attribute if (appliedAttribute[0] !== 'author' || appliedAttribute[1] !== '') { @@ -376,27 +373,19 @@ PadDiff.prototype._createDeletionChangeset = function (cs, startAText, apool) { // If the text this operator applies to is only a star, // than this is a false positive and should be ignored if (csOp.attribs && textBank !== '*') { - const deletedAttrib = apool.putAttrib(['removed', true]); - let authorAttrib = apool.putAttrib(['author', '']); - const attribs = []; - Changeset.eachAttribNumber(csOp.attribs, (n) => { - const attrib = apool.getAttrib(n); - attribs.push(attrib); - if (attrib[0] === 'author') authorAttrib = n; - }); + const attribs = AttributeMap.fromString(csOp.attribs, apool); const undoBackToAttribs = cachedStrFunc((oldAttribsStr) => { - const backAttribs = []; + const oldAttribs = AttributeMap.fromString(oldAttribsStr, apool); + const backAttribs = new AttributeMap(apool) + .set('author', '') + .set('removed', 'true'); for (const [key, value] of attribs) { - const oldValue = Changeset.attribsAttributeValue(oldAttribsStr, key, apool); - if (oldValue !== value) backAttribs.push([key, oldValue]) + const oldValue = oldAttribs.get(key); + if (oldValue !== value) backAttribs.set(key, oldValue); } - - return Changeset.makeAttribsString('=', backAttribs, apool); + return backAttribs.toString(); }); - const oldAttribsAddition = - `*${Changeset.numToString(deletedAttrib)}*${Changeset.numToString(authorAttrib)}`; - let textLeftToProcess = textBank; while (textLeftToProcess.length > 0) { @@ -429,7 +418,7 @@ PadDiff.prototype._createDeletionChangeset = function (cs, startAText, apool) { let textBankIndex = 0; consumeAttribRuns(lengthToProcess, (len, attribs, endsLine) => { // get the old attributes back - const oldAttribs = (undoBackToAttribs(attribs) || '') + oldAttribsAddition; + const oldAttribs = undoBackToAttribs(attribs); builder.insert(processText.substr(textBankIndex, len), oldAttribs); textBankIndex += len; diff --git a/src/static/js/AttributeManager.js b/src/static/js/AttributeManager.js index 760422427..ebef74c07 100644 --- a/src/static/js/AttributeManager.js +++ b/src/static/js/AttributeManager.js @@ -1,7 +1,9 @@ 'use strict'; +const AttributeMap = require('./AttributeMap'); const Changeset = require('./Changeset'); const ChangesetUtils = require('./ChangesetUtils'); +const attributes = require('./attributes'); const _ = require('./underscore'); const lineMarkerAttribute = 'lmkr'; @@ -150,7 +152,7 @@ AttributeManager.prototype = _(AttributeManager.prototype).extend({ if (!aline) return ''; const opIter = Changeset.opIterator(aline); if (!opIter.hasNext()) return ''; - return Changeset.opAttributeValue(opIter.next(), attributeName, this.rep.apool) || ''; + return AttributeMap.fromString(opIter.next().attribs, this.rep.apool).get(attributeName) || ''; }, /* @@ -164,10 +166,7 @@ AttributeManager.prototype = _(AttributeManager.prototype).extend({ const opIter = Changeset.opIterator(aline); if (!opIter.hasNext()) return []; const op = opIter.next(); - if (!op.attribs) return []; - const attributes = []; - Changeset.eachAttribNumber(op.attribs, (n) => attributes.push(this.rep.apool.getAttrib(n))); - return attributes; + return [...attributes.attribsFromString(op.attribs, this.rep.apool)]; }, /* @@ -191,9 +190,7 @@ AttributeManager.prototype = _(AttributeManager.prototype).extend({ } } - const withIt = Changeset.makeAttribsString('+', [ - [attributeName, 'true'], - ], rep.apool); + const withIt = new AttributeMap(rep.apool).set(attributeName, 'true').toString(); const withItRegex = new RegExp(`${withIt.replace(/\*/g, '\\*')}(\\*|$)`); const hasIt = (attribs) => withItRegex.test(attribs); @@ -274,10 +271,7 @@ AttributeManager.prototype = _(AttributeManager.prototype).extend({ currentOperation = opIter.next(); currentPointer += currentOperation.chars; if (currentPointer <= column) continue; - const attributes = []; - Changeset.eachAttribNumber( - currentOperation.attribs, (n) => attributes.push(this.rep.apool.getAttrib(n))); - return attributes; + return [...attributes.attribsFromString(currentOperation.attribs, this.rep.apool)]; } return []; }, diff --git a/src/static/js/Changeset.js b/src/static/js/Changeset.js index 71d7fe115..11494a5f7 100644 --- a/src/static/js/Changeset.js +++ b/src/static/js/Changeset.js @@ -22,7 +22,9 @@ * https://github.com/ether/pad/blob/master/infrastructure/ace/www/easysync2.js */ +const AttributeMap = require('./AttributeMap'); const AttributePool = require('./AttributePool'); +const attributes = require('./attributes'); const {padutils} = require('./pad_utils'); /** @@ -31,6 +33,15 @@ const {padutils} = require('./pad_utils'); * @typedef {[string, string]} Attribute */ +/** + * A concatenated sequence of zero or more attribute identifiers, each one represented by an + * asterisk followed by a base-36 encoded attribute number. + * + * Examples: '', '*0', '*3*j*z*1q' + * + * @typedef {string} AttributeString + */ + /** * This method is called whenever there is an error in the sync process. * @@ -236,15 +247,19 @@ const copyOp = (op1, op2 = exports.newOp()) => Object.assign(op2, op1); * * @param {('-'|'+'|'=')} opcode - The operator to use. * @param {string} text - The text to remove/add/keep. - * @param {(string|Attribute[])} [attribs] - The attributes to apply to the operations. See - * `makeAttribsString`. - * @param {?AttributePool} [pool] - See `makeAttribsString`. + * @param {(Iterable|AttributeString)} [attribs] - The attributes to insert into the pool + * (if necessary) and encode. If an attribute string, no checking is performed to ensure that + * the attributes exist in the pool, are in the canonical order, and contain no duplicate keys. + * If this is an iterable of attributes, `pool` must be non-null. + * @param {?AttributePool} pool - Attribute pool. Required if `attribs` is an iterable of + * attributes, ignored if `attribs` is an attribute string. * @yields {Op} One or two ops (depending on the presense of newlines) that cover the given text. * @returns {Generator} */ const opsFromText = function* (opcode, text, attribs = '', pool = null) { const op = exports.newOp(opcode); - op.attribs = exports.makeAttribsString(opcode, attribs, pool); + op.attribs = typeof attribs === 'string' + ? attribs : new AttributeMap(pool).update(attribs || [], opcode === '+').toString(); const lastNewlinePos = text.lastIndexOf('\n'); if (lastNewlinePos < 0) { op.chars = text.length; @@ -387,9 +402,9 @@ exports.smartOpAssembler = () => { * @deprecated Use `opsFromText` instead. * @param {('-'|'+'|'=')} opcode - The operator to use. * @param {string} text - The text to remove/add/keep. - * @param {(string|Attribute[])} attribs - The attributes to apply to the operations. See - * `makeAttribsString`. - * @param {?AttributePool} pool - See `makeAttribsString`. + * @param {(string|Iterable)} attribs - The attributes to apply to the operations. + * @param {?AttributePool} pool - Attribute pool. Only required if `attribs` is an iterable of + * attribute key, value pairs. */ const appendOpWithText = (opcode, text, attribs, pool) => { padutils.warnWithStack('Changeset.smartOpAssembler().appendOpWithText() is deprecated; ' + @@ -1109,20 +1124,11 @@ exports.mutateTextLines = (cs, lines) => { mut.close(); }; -/** - * Sorts an array of attributes by key. - * - * @param {Attribute[]} attribs - The array of attributes to sort in place. - * @returns {Attribute[]} The `attribs` array. - */ -const sortAttribs = - (attribs) => attribs.sort((a, b) => (a[0] > b[0] ? 1 : 0) - (a[0] < b[0] ? 1 : 0)); - /** * Composes two attribute strings (see below) into one. * - * @param {string} att1 - first attribute string - * @param {string} att2 - second attribue string + * @param {AttributeString} att1 - first attribute string + * @param {AttributeString} att2 - second attribue string * @param {boolean} resultIsMutation - * @param {AttributePool} pool - attribute pool * @returns {string} @@ -1149,27 +1155,7 @@ exports.composeAttributes = (att1, att2, resultIsMutation, pool) => { return att2; } if (!att2) return att1; - const atts = new Map(); - att1.replace(/\*([0-9a-z]+)/g, (_, a) => { - const [key, val] = pool.getAttrib(exports.parseNum(a)); - atts.set(key, val); - return ''; - }); - att2.replace(/\*([0-9a-z]+)/g, (_, a) => { - const [key, val] = pool.getAttrib(exports.parseNum(a)); - if (val || resultIsMutation) { - atts.set(key, val); - } else { - atts.delete(key); - } - return ''; - }); - const buf = exports.stringAssembler(); - for (const att of sortAttribs([...atts])) { - buf.append('*'); - buf.append(exports.numToString(pool.putAttrib(att))); - } - return buf.toString(); + return AttributeMap.fromString(att1, pool).updateFromString(att2, !resultIsMutation).toString(); }; /** @@ -1611,16 +1597,21 @@ exports.makeAttribution = (text) => { * Iterates over attributes in exports, attribution string, or attribs property of an op and runs * function func on them. * + * @deprecated Use `attributes.decodeAttribString()` instead. * @param {string} cs - changeset * @param {Function} func - function to call */ exports.eachAttribNumber = (cs, func) => { + padutils.warnWithStack('Changeset.eachAttribNumber() is deprecated; ' + + 'use attributes.decodeAttribString() instead'); let dollarPos = cs.indexOf('$'); if (dollarPos < 0) { dollarPos = cs.length; } const upToDollar = cs.substring(0, dollarPos); + // WARNING: The following cannot be replaced with a call to `attributes.decodeAttribString()` + // because that function only works on attribute strings, not serialized operations or changesets. upToDollar.replace(/\*([0-9a-z]+)/g, (_, a) => { func(exports.parseNum(a)); return ''; @@ -1784,33 +1775,44 @@ exports.isIdentity = (cs) => { return unpacked.ops === '' && unpacked.oldLen === unpacked.newLen; }; +/** + * @deprecated Use an AttributeMap instead. + */ +const attribsAttributeValue = (attribs, key, pool) => { + if (!attribs) return ''; + for (const [k, v] of attributes.attribsFromString(attribs, pool)) { + if (k === key) return v; + } + return ''; +}; + /** * Returns all the values of attributes with a certain key in an Op attribs string. * + * @deprecated Use an AttributeMap instead. * @param {Op} op - Op * @param {string} key - string to search for * @param {AttributePool} pool - attribute pool * @returns {string} */ -exports.opAttributeValue = (op, key, pool) => exports.attribsAttributeValue(op.attribs, key, pool); +exports.opAttributeValue = (op, key, pool) => { + padutils.warnWithStack('Changeset.opAttributeValue() is deprecated; use an AttributeMap instead'); + return attribsAttributeValue(op.attribs, key, pool); +}; /** * Returns all the values of attributes with a certain key in an attribs string. * - * @param {string} attribs - Attribute string + * @deprecated Use an AttributeMap instead. + * @param {AttributeString} attribs - Attribute string * @param {string} key - string to search for * @param {AttributePool} pool - attribute pool * @returns {string} */ exports.attribsAttributeValue = (attribs, key, pool) => { - if (!attribs) return ''; - let value = ''; - exports.eachAttribNumber(attribs, (n) => { - if (pool.getAttribKey(n) === key) { - value = pool.getAttribValue(n); - } - }); - return value; + padutils.warnWithStack('Changeset.attribsAttributeValue() is deprecated; ' + + 'use an AttributeMap instead'); + return attribsAttributeValue(attribs, key, pool); }; /** @@ -1846,7 +1848,8 @@ exports.builder = (oldLen) => { */ keep: (N, L, attribs, pool) => { o.opcode = '='; - o.attribs = (attribs && exports.makeAttribsString('=', attribs, pool)) || ''; + o.attribs = typeof attribs === 'string' + ? attribs : new AttributeMap(pool).update(attribs || []).toString(); o.chars = N; o.lines = (L || 0); assem.append(o); @@ -1908,8 +1911,9 @@ exports.builder = (oldLen) => { /** * Constructs an attribute string from a sequence of attributes. * + * @deprecated Use `AttributeMap.prototype.toString()` or `attributes.attribsToString()` instead. * @param {string} opcode - The opcode for the Op that will get the resulting attribute string. - * @param {?(Attribute[]|AttributeString)} attribs - The attributes to insert into the pool + * @param {?(Iterable|AttributeString)} attribs - The attributes to insert into the pool * (if necessary) and encode. If an attribute string, no checking is performed to ensure that * the attributes exist in the pool, are in the canonical order, and contain no duplicate keys. * If this is an iterable of attributes, `pool` must be non-null. @@ -1918,11 +1922,12 @@ exports.builder = (oldLen) => { * @returns {AttributeString} */ exports.makeAttribsString = (opcode, attribs, pool) => { + padutils.warnWithStack( + 'Changeset.makeAttribsString() is deprecated; ' + + 'use AttributeMap.prototype.toString() or attributes.attribsToString() instead'); if (!attribs || !['=', '+'].includes(opcode)) return ''; if (typeof attribs === 'string') return attribs; - return sortAttribs(attribs.filter(([k, v]) => v || opcode === '=')) - .map((a) => `*${exports.numToString(pool.putAttrib(a))}`) - .join(''); + return new AttributeMap(pool).update(attribs, opcode === '+').toString(); }; /** @@ -2085,17 +2090,15 @@ exports.inverse = (cs, lines, alines, pool) => { const csOp = csIter.next(); if (csOp.opcode === '=') { if (csOp.attribs) { - const csAttribs = []; - exports.eachAttribNumber(csOp.attribs, (n) => csAttribs.push(pool.getAttrib(n))); - const undoBackToAttribs = cachedStrFunc((attribs) => { - const backAttribs = []; - for (const [appliedKey, appliedValue] of csAttribs) { - const oldValue = exports.attribsAttributeValue(attribs, appliedKey, pool); - if (appliedValue !== oldValue) { - backAttribs.push([appliedKey, oldValue]); - } + const attribs = AttributeMap.fromString(csOp.attribs, pool); + const undoBackToAttribs = cachedStrFunc((oldAttribsStr) => { + const oldAttribs = AttributeMap.fromString(oldAttribsStr, pool); + const backAttribs = new AttributeMap(pool); + for (const [key, value] of attribs) { + const oldValue = oldAttribs.get(key) || ''; + if (oldValue !== value) backAttribs.set(key, oldValue); } - return exports.makeAttribsString('=', backAttribs, pool); + return backAttribs.toString(); }); consumeAttribRuns(csOp.chars, (len, attribs, endsLine) => { builder.keep(len, endsLine ? 1 : 0, undoBackToAttribs(attribs)); diff --git a/src/static/js/ace2_inner.js b/src/static/js/ace2_inner.js index 675427019..46c6ca579 100644 --- a/src/static/js/ace2_inner.js +++ b/src/static/js/ace2_inner.js @@ -18,6 +18,7 @@ */ let documentAttributeManager; +const AttributeMap = require('./AttributeMap'); const browser = require('./vendors/browser'); const padutils = require('./pad_utils').padutils; const Ace2Common = require('./ace2_common'); @@ -1542,9 +1543,7 @@ function Ace2Inner(editorInfo, cssManagers) { } } - const withIt = Changeset.makeAttribsString('+', [ - [attributeName, 'true'], - ], rep.apool); + const withIt = new AttributeMap(rep.apool).set(attributeName, 'true').toString(); const withItRegex = new RegExp(`${withIt.replace(/\*/g, '\\*')}(\\*|$)`); const hasIt = (attribs) => withItRegex.test(attribs); @@ -1608,9 +1607,7 @@ function Ace2Inner(editorInfo, cssManagers) { if (!(rep.selStart && rep.selEnd)) return; let selectionAllHasIt = true; - const withIt = Changeset.makeAttribsString('+', [ - [attributeName, 'true'], - ], rep.apool); + const withIt = new AttributeMap(rep.apool).set(attributeName, 'true').toString(); const withItRegex = new RegExp(`${withIt.replace(/\*/g, '\\*')}(\\*|$)`); const hasIt = (attribs) => withItRegex.test(attribs); @@ -1820,22 +1817,15 @@ function Ace2Inner(editorInfo, cssManagers) { } let isNewTextMultiauthor = false; - const authorAtt = Changeset.makeAttribsString('+', (thisAuthor ? [ - ['author', thisAuthor], - ] : []), rep.apool); const authorizer = cachedStrFunc((oldAtts) => { - if (isNewTextMultiauthor) { - // prefer colors from DOM - return Changeset.composeAttributes(authorAtt, oldAtts, true, rep.apool); - } else { - // use this author's color - return Changeset.composeAttributes(oldAtts, authorAtt, true, rep.apool); - } + const attribs = AttributeMap.fromString(oldAtts, rep.apool); + if (!isNewTextMultiauthor || !attribs.has('author')) attribs.set('author', thisAuthor); + return attribs.toString(); }); let foundDomAuthor = ''; eachAttribRun(newAttribs, (start, end, attribs) => { - const a = Changeset.attribsAttributeValue(attribs, 'author', rep.apool); + const a = AttributeMap.fromString(attribs, rep.apool).get('author'); if (a && a !== foundDomAuthor) { if (!foundDomAuthor) { foundDomAuthor = a; @@ -2632,8 +2622,8 @@ function Ace2Inner(editorInfo, cssManagers) { const opIter = Changeset.opIterator(alineAttrs); while (opIter.hasNext()) { const op = opIter.next(); - const authorId = Changeset.opAttributeValue(op, 'author', apool); - if (authorId !== '') authorIds.add(authorId); + const authorId = AttributeMap.fromString(op.attribs, apool).get('author'); + if (authorId) authorIds.add(authorId); } } const idToName = new Map(parent.parent.pad.userList().map((a) => [a.userId, a.name])); diff --git a/src/static/js/broadcast.js b/src/static/js/broadcast.js index c778f6909..5b19acf88 100644 --- a/src/static/js/broadcast.js +++ b/src/static/js/broadcast.js @@ -26,6 +26,7 @@ const makeCSSManager = require('./cssmanager').makeCSSManager; const domline = require('./domline').domline; const AttribPool = require('./AttributePool'); const Changeset = require('./Changeset'); +const attributes = require('./attributes'); const linestylefilter = require('./linestylefilter').linestylefilter; const colorutils = require('./colorutils').colorutils; const _ = require('./underscore'); @@ -114,20 +115,18 @@ const loadBroadcastJS = (socket, sendSocketMsg, fireWhenAllScriptsAreLoaded, Bro }, getActiveAuthors() { - const authors = []; - const seenNums = {}; - const alines = this.alines; - for (let i = 0; i < alines.length; i++) { - Changeset.eachAttribNumber(alines[i], (n) => { - if (seenNums[n]) return; - seenNums[n] = true; - if (this.apool.getAttribKey(n) !== 'author') return; - const a = this.apool.getAttribValue(n); - if (a) authors.push(a); - }); + const authorIds = new Set(); + for (const aline of this.alines) { + const opIter = Changeset.opIterator(aline); + while (opIter.hasNext()) { + const op = opIter.next(); + for (const [k, v] of attributes.attribsFromString(op.attribs, this.apool)) { + if (k !== 'author') continue; + if (v) authorIds.add(v); + } + } } - authors.sort(); - return authors; + return [...authorIds].sort(); }, }; diff --git a/src/static/js/changesettracker.js b/src/static/js/changesettracker.js index cfb6c88a3..c45a253d4 100644 --- a/src/static/js/changesettracker.js +++ b/src/static/js/changesettracker.js @@ -22,6 +22,7 @@ * limitations under the License. */ +const AttributeMap = require('./AttributeMap'); const AttributePool = require('./AttributePool'); const Changeset = require('./Changeset'); @@ -141,7 +142,6 @@ const makeChangesetTracker = (scheduler, apool, aceCallbacksProvider) => { // Sanitize authorship: Replace all author attributes with this user's author ID in case the // text was copied from another author. - const authorAttr = Changeset.numToString(apool.putAttrib(['author', authorId])); const cs = Changeset.unpack(userChangeset); const iterator = Changeset.opIterator(cs.ops); let op; @@ -150,18 +150,12 @@ const makeChangesetTracker = (scheduler, apool, aceCallbacksProvider) => { while (iterator.hasNext()) { op = iterator.next(); if (op.opcode === '+') { - let newAttrs = ''; - - op.attribs.split('*').forEach((attrNum) => { - if (!attrNum) return; - const attr = apool.getAttrib(parseInt(attrNum, 36)); - if (!attr) return; - if ('author' === attr[0]) { - // replace that author with the current one - newAttrs += `*${authorAttr}`; - } else { newAttrs += `*${attrNum}`; } // overtake all other attribs as is - }); - op.attribs = newAttrs; + const attribs = AttributeMap.fromString(op.attribs, apool); + const oldAuthorId = attribs.get('author'); + if (oldAuthorId != null && oldAuthorId !== authorId) { + attribs.set('author', authorId); + op.attribs = attribs.toString(); + } } assem.append(op); } diff --git a/src/static/js/contentcollector.js b/src/static/js/contentcollector.js index 54c628804..59e726232 100644 --- a/src/static/js/contentcollector.js +++ b/src/static/js/contentcollector.js @@ -26,6 +26,7 @@ const _MAX_LIST_LEVEL = 16; +const AttributeMap = require('./AttributeMap'); const UNorm = require('unorm'); const Changeset = require('./Changeset'); const hooks = require('./pluginfw/hooks'); @@ -227,7 +228,7 @@ const makeContentCollector = (collectStyles, abrowser, apool, className2Author) }; const _recalcAttribString = (state) => { - const lst = []; + const attribs = new AttributeMap(apool); for (const [a, count] of Object.entries(state.attribs)) { if (!count) continue; // The following splitting of the attribute name is a workaround @@ -241,32 +242,31 @@ const makeContentCollector = (collectStyles, abrowser, apool, className2Author) if (attributeSplits.length > 1) { // the attribute name follows the convention key::value // so save it as a key value attribute - lst.push([attributeSplits[0], attributeSplits[1]]); + const [k, v] = attributeSplits; + if (v) attribs.set(k, v); } else { // the "normal" case, the attribute is just a switch // so set it true - lst.push([a, 'true']); + attribs.set(a, 'true'); } } if (state.authorLevel > 0) { - const authorAttrib = ['author', state.author]; - if (apool.putAttrib(authorAttrib, true) >= 0) { + if (apool.putAttrib(['author', state.author], true) >= 0) { // require that author already be in pool // (don't add authors from other documents, etc.) - lst.push(authorAttrib); + if (state.author) attribs.set('author', state.author); } } - state.attribString = Changeset.makeAttribsString('+', lst, apool); + state.attribString = attribs.toString(); }; const _produceLineAttributesMarker = (state) => { // TODO: This has to go to AttributeManager. - const attributes = [ - ['lmkr', '1'], - ['insertorder', 'first'], - ...Object.entries(state.lineAttributes), - ]; - lines.appendText('*', Changeset.makeAttribsString('+', attributes, apool)); + const attribs = new AttributeMap(apool) + .set('lmkr', '1') + .set('insertorder', 'first') + .update(Object.entries(state.lineAttributes).map(([k, v]) => [k, v || '']), true); + lines.appendText('*', attribs.toString()); }; cc.startNewLine = (state) => { if (state) { diff --git a/src/static/js/linestylefilter.js b/src/static/js/linestylefilter.js index 0821889c7..f6bcbb54e 100644 --- a/src/static/js/linestylefilter.js +++ b/src/static/js/linestylefilter.js @@ -31,6 +31,7 @@ // requires: undefined const Changeset = require('./Changeset'); +const attributes = require('./attributes'); const hooks = require('./pluginfw/hooks'); const linestylefilter = {}; const AttributeManager = require('./AttributeManager'); @@ -73,10 +74,8 @@ linestylefilter.getLineStyleFilter = (lineLength, aline, textAndClassFunc, apool let classes = ''; let isLineAttribMarker = false; - // For each attribute number - Changeset.eachAttribNumber(attribs, (n) => { - const [key, value] = apool.getAttrib(n); - if (!key || !value) return; + for (const [key, value] of attributes.attribsFromString(attribs, apool)) { + if (!key || !value) continue; if (!isLineAttribMarker && AttributeManager.lineAttributes.indexOf(key) >= 0) { isLineAttribMarker = true; } @@ -93,7 +92,7 @@ linestylefilter.getLineStyleFilter = (lineLength, aline, textAndClassFunc, apool const results = hooks.callAll('aceAttribsToClasses', {linestylefilter, key, value}); classes += ` ${results.join(' ')}`; } - }); + } if (isLineAttribMarker) classes += ` ${lineAttributeMarker}`; return classes.substring(1); diff --git a/src/tests/backend/specs/contentcollector.js b/src/tests/backend/specs/contentcollector.js index a6ff8c2d7..c2cee5338 100644 --- a/src/tests/backend/specs/contentcollector.js +++ b/src/tests/backend/specs/contentcollector.js @@ -12,6 +12,7 @@ const AttributePool = require('../../../static/js/AttributePool'); const Changeset = require('../../../static/js/Changeset'); const assert = require('assert').strict; +const attributes = require('../../../static/js/attributes'); const contentcollector = require('../../../static/js/contentcollector'); const jsdom = require('jsdom'); @@ -348,7 +349,9 @@ describe(__filename, function () { const opIter = Changeset.opIterator(aline); while (opIter.hasNext()) { const op = opIter.next(); - Changeset.eachAttribNumber(op.attribs, (n) => assert(n < knownAttribs.length)); + for (const n of attributes.decodeAttribString(op.attribs)) { + assert(n < knownAttribs.length); + } } } const cc = contentcollector.makeContentCollector(true, null, apool); @@ -375,16 +378,9 @@ describe(__filename, function () { const opIter = Changeset.opIterator(aline); while (opIter.hasNext()) { const op = opIter.next(); - const gotOpAttribs = []; + const gotOpAttribs = [...attributes.attribsFromString(op.attribs, apool)]; gotAlineAttribs.push(gotOpAttribs); - const wantOpAttribs = []; - wantAlineAttribs.push(wantOpAttribs); - Changeset.eachAttribNumber(op.attribs, (n) => { - const attrib = apool.getAttrib(n); - gotOpAttribs.push(attrib); - wantOpAttribs.push(attrib); - }); - wantOpAttribs.sort(([keyA], [keyB]) => (keyA > keyB ? 1 : 0) - (keyA < keyB ? 1 : 0)); + wantAlineAttribs.push(attributes.sort([...gotOpAttribs])); } } assert.deepEqual(gotAttribs, wantAttribs); From 2fc06a08842cc7d5f7ef105ed96f1faea2da0551 Mon Sep 17 00:00:00 2001 From: Richard Hansen Date: Fri, 19 Nov 2021 01:10:04 -0500 Subject: [PATCH 017/446] Changeset: Add TODO comments for issues noticed --- src/node/utils/padDiff.js | 2 ++ src/static/js/Changeset.js | 2 ++ src/static/js/contentcollector.js | 6 ++++++ 3 files changed, 10 insertions(+) diff --git a/src/node/utils/padDiff.js b/src/node/utils/padDiff.js index a8841cf22..05cba308f 100644 --- a/src/node/utils/padDiff.js +++ b/src/node/utils/padDiff.js @@ -383,6 +383,8 @@ PadDiff.prototype._createDeletionChangeset = function (cs, startAText, apool) { const oldValue = oldAttribs.get(key); if (oldValue !== value) backAttribs.set(key, oldValue); } + // TODO: backAttribs does not restore removed attributes (it is missing attributes that + // are in oldAttribs but not in attribs). I don't know if that is intentional. return backAttribs.toString(); }); diff --git a/src/static/js/Changeset.js b/src/static/js/Changeset.js index 11494a5f7..24cc855a2 100644 --- a/src/static/js/Changeset.js +++ b/src/static/js/Changeset.js @@ -2098,6 +2098,8 @@ exports.inverse = (cs, lines, alines, pool) => { const oldValue = oldAttribs.get(key) || ''; if (oldValue !== value) backAttribs.set(key, oldValue); } + // TODO: backAttribs does not restore removed attributes (it is missing attributes that + // are in oldAttribs but not in attribs). I don't know if that is intentional. return backAttribs.toString(); }); consumeAttribRuns(csOp.chars, (len, attribs, endsLine) => { diff --git a/src/static/js/contentcollector.js b/src/static/js/contentcollector.js index 59e726232..2e6005bd8 100644 --- a/src/static/js/contentcollector.js +++ b/src/static/js/contentcollector.js @@ -235,6 +235,9 @@ const makeContentCollector = (collectStyles, abrowser, apool, className2Author) // to enable the content collector to store key-value attributes // see https://github.com/ether/etherpad-lite/issues/2567 for more information // in long term the contentcollector should be refactored to get rid of this workaround + // + // TODO: This approach doesn't support changing existing values: if both 'foo::bar' and + // 'foo::baz' are in state.attribs then the last one encountered while iterating will win. const ATTRIBUTE_SPLIT_STRING = '::'; // see if attributeString is splittable @@ -265,6 +268,9 @@ const makeContentCollector = (collectStyles, abrowser, apool, className2Author) const attribs = new AttributeMap(apool) .set('lmkr', '1') .set('insertorder', 'first') + // TODO: Converting all falsy values in state.lineAttributes into removals is awkward. + // Better would be to never add 0, false, null, or undefined to state.lineAttributes in the + // first place (I'm looking at you, state.lineAttributes.start). .update(Object.entries(state.lineAttributes).map(([k, v]) => [k, v || '']), true); lines.appendText('*', attribs.toString()); }; From 9e772df9917adab7f21afaf706dc7e784c80e9e7 Mon Sep 17 00:00:00 2001 From: Richard Hansen Date: Sun, 21 Nov 2021 03:38:15 -0500 Subject: [PATCH 018/446] deps: Bump dependencies --- src/package-lock.json | 742 +++++++++++++++++++++++------------------- src/package.json | 26 +- 2 files changed, 418 insertions(+), 350 deletions(-) diff --git a/src/package-lock.json b/src/package-lock.json index fd7e6d3cb..594907f71 100644 --- a/src/package-lock.json +++ b/src/package-lock.json @@ -38,9 +38,9 @@ } }, "@azure/core-client": { - "version": "1.3.1", - "resolved": "https://registry.npmjs.org/@azure/core-client/-/core-client-1.3.1.tgz", - "integrity": "sha512-7IHm2DGg2u7dJYtCW84Ik7uENHfE8VsM/sWloZezPKYDoWZrg7JzwjvdGAfsaELKi2p0GE+JBaAbDYnNpr5V1w==", + "version": "1.3.2", + "resolved": "https://registry.npmjs.org/@azure/core-client/-/core-client-1.3.2.tgz", + "integrity": "sha512-qfkRYKmeEmisluMdGTbBtXeyBLaImjFeVW0gcT5yRAwxJmlnTvSyD+a3PjukAtjIrl/tnb4WSJOBpONSJ91+5Q==", "requires": { "@azure/abort-controller": "^1.0.0", "@azure/core-asynciterator-polyfill": "^1.0.0", @@ -62,9 +62,9 @@ } }, "@azure/core-http": { - "version": "2.2.1", - "resolved": "https://registry.npmjs.org/@azure/core-http/-/core-http-2.2.1.tgz", - "integrity": "sha512-7ATnV3OGzCO2K9kMrh3NKUM8b4v+xasmlUhkNZz6uMbm+8XH/AexLkhRGsoo0GyKNlEGvyGEfytqTk0nUY2I4A==", + "version": "2.2.2", + "resolved": "https://registry.npmjs.org/@azure/core-http/-/core-http-2.2.2.tgz", + "integrity": "sha512-V1DdoO9V/sFimKpdWoNBgsE+QUjQgpXYnxrTdUp5RyhsTJjvEVn/HKmTQXIHuLUUo6IyIWj+B+Dg4VaXse9dIA==", "requires": { "@azure/abort-controller": "^1.0.0", "@azure/core-asynciterator-polyfill": "^1.0.0", @@ -131,9 +131,9 @@ } }, "@azure/core-rest-pipeline": { - "version": "1.3.1", - "resolved": "https://registry.npmjs.org/@azure/core-rest-pipeline/-/core-rest-pipeline-1.3.1.tgz", - "integrity": "sha512-xTQiv47O5cWzJFkwiDrUTT4K4IYbUIts0gaou5TZxAAuhQi9kAKWHEmFTjHVMOeAmyDhlMM5cb21M2n4WDto1A==", + "version": "1.3.2", + "resolved": "https://registry.npmjs.org/@azure/core-rest-pipeline/-/core-rest-pipeline-1.3.2.tgz", + "integrity": "sha512-kymICKESeHBpVLgQiAxllgWdSTopkqtmfPac8ITwMCxNEC6hzbSpqApYbjzxbBNkBMgoD4GESo6LLhR/sPh6kA==", "requires": { "@azure/abort-controller": "^1.0.0", "@azure/core-auth": "^1.3.0", @@ -521,20 +521,87 @@ "integrity": "sha512-GaHYm+c0O9MjZRu0ongGBRbinu8gVAMd2UZjji6jVmqKtZluZnptXGWhz1E8j8D2HJ3f/yMxKAUC0b+57wncIw==" }, "@mapbox/node-pre-gyp": { - "version": "1.0.5", - "resolved": "https://registry.npmjs.org/@mapbox/node-pre-gyp/-/node-pre-gyp-1.0.5.tgz", - "integrity": "sha512-4srsKPXWlIxp5Vbqz5uLfBN+du2fJChBoYn/f2h991WLdk7jUvcSk/McVLSv/X+xQIPI8eGD5GjrnygdyHnhPA==", + "version": "1.0.7", + "resolved": "https://registry.npmjs.org/@mapbox/node-pre-gyp/-/node-pre-gyp-1.0.7.tgz", + "integrity": "sha512-PplSvl4pJ5N3BkVjAdDzpPhVUPdC73JgttkR+LnBx2OORC1GCQsBjUeEuipf9uOaAM1SbxcdZFfR3KDTKm2S0A==", "optional": true, "requires": { "detect-libc": "^1.0.3", "https-proxy-agent": "^5.0.0", "make-dir": "^3.1.0", - "node-fetch": "^2.6.1", + "node-fetch": "^2.6.5", "nopt": "^5.0.0", - "npmlog": "^4.1.2", + "npmlog": "^6.0.0", "rimraf": "^3.0.2", - "semver": "^7.3.4", - "tar": "^6.1.0" + "semver": "^7.3.5", + "tar": "^6.1.11" + }, + "dependencies": { + "are-we-there-yet": { + "version": "2.0.0", + "resolved": "https://registry.npmjs.org/are-we-there-yet/-/are-we-there-yet-2.0.0.tgz", + "integrity": "sha512-Ci/qENmwHnsYo9xKIcUJN5LeDKdJ6R1Z1j9V/J5wyq8nh/mYPEpIKJbBZXtZjG04HiK7zV/p6Vs9952MrMeUIw==", + "optional": true, + "requires": { + "delegates": "^1.0.0", + "readable-stream": "^3.6.0" + } + }, + "gauge": { + "version": "4.0.0", + "resolved": "https://registry.npmjs.org/gauge/-/gauge-4.0.0.tgz", + "integrity": "sha512-F8sU45yQpjQjxKkm1UOAhf0U/O0aFt//Fl7hsrNVto+patMHjs7dPI9mFOGUKbhrgKm0S3EjW3scMFuQmWSROw==", + "optional": true, + "requires": { + "ansi-regex": "^5.0.1", + "aproba": "^1.0.3 || ^2.0.0", + "color-support": "^1.1.2", + "console-control-strings": "^1.0.0", + "has-unicode": "^2.0.1", + "signal-exit": "^3.0.0", + "string-width": "^4.2.3", + "strip-ansi": "^6.0.1", + "wide-align": "^1.1.2" + } + }, + "npmlog": { + "version": "6.0.0", + "resolved": "https://registry.npmjs.org/npmlog/-/npmlog-6.0.0.tgz", + "integrity": "sha512-03ppFRGlsyUaQFbGC2C8QWJN/C/K7PsfyD9aQdhVKAQIH4sQBc8WASqFBP7O+Ut4d2oo5LoeoboB3cGdBZSp6Q==", + "optional": true, + "requires": { + "are-we-there-yet": "^2.0.0", + "console-control-strings": "^1.1.0", + "gauge": "^4.0.0", + "set-blocking": "^2.0.0" + } + }, + "readable-stream": { + "version": "3.6.0", + "resolved": "https://registry.npmjs.org/readable-stream/-/readable-stream-3.6.0.tgz", + "integrity": "sha512-BViHy7LKeTz4oNnkcLJ+lVSL6vpiFeX6/d3oSH8zCW7UxP2onchk+vTGB143xuFjHS3deTgkKoXXymXqymiIdA==", + "optional": true, + "requires": { + "inherits": "^2.0.3", + "string_decoder": "^1.1.1", + "util-deprecate": "^1.0.1" + } + }, + "safe-buffer": { + "version": "5.2.1", + "resolved": "https://registry.npmjs.org/safe-buffer/-/safe-buffer-5.2.1.tgz", + "integrity": "sha512-rp3So07KcdmmKbGvgaNxQSJr7bGVSVk5S9Eq1F+ppbRo70+YeaDxkw5Dd8NPN+GD6bjnYm2VuPuCXmpuYvmCXQ==", + "optional": true + }, + "string_decoder": { + "version": "1.3.0", + "resolved": "https://registry.npmjs.org/string_decoder/-/string_decoder-1.3.0.tgz", + "integrity": "sha512-hkRX8U1WjJFd8LsDJ2yQ/wWWxaopEsABU1XfkM8A+j0+85JAGppt16cr1Whg6KIbb4okU6Mql6BOj+uup/wKeA==", + "optional": true, + "requires": { + "safe-buffer": "~5.2.0" + } + } } }, "@opentelemetry/api": { @@ -552,9 +619,9 @@ } }, "@sinonjs/fake-timers": { - "version": "7.1.2", - "resolved": "https://registry.npmjs.org/@sinonjs/fake-timers/-/fake-timers-7.1.2.tgz", - "integrity": "sha512-iQADsW4LBMISqZ6Ci1dupJL9pprqwcVFTcOsEmQOEhW+KLCVn/Y4Jrvg2k19fIHCp+iFprriYPTdRcQR8NbUPg==", + "version": "8.1.0", + "resolved": "https://registry.npmjs.org/@sinonjs/fake-timers/-/fake-timers-8.1.0.tgz", + "integrity": "sha512-OAPJUAtgeINhh/TAlUID4QTs53Njm7xzddaVlEs/SXwgtiD1tW22zAB/W1wdqfrpmikgaWQ9Fw6Ws+hsiRm5Vg==", "dev": true, "requires": { "@sinonjs/commons": "^1.7.0" @@ -606,9 +673,9 @@ "integrity": "sha512-5tXH6Bx/kNGd3MgffdmP4dy2Z+G4eaXw0SE81Tq3BNadtnMR5/ySMzX4SLEzHJzSmPNn4HIdpQsBvXMUykr58w==" }, "@types/node": { - "version": "16.10.3", - "resolved": "https://registry.npmjs.org/@types/node/-/node-16.10.3.tgz", - "integrity": "sha512-ho3Ruq+fFnBrZhUYI46n/bV2GjwzSkwuT4dTf0GkuNFmnb8nq4ny2z9JEVemFi6bdEJanHLlYfy9c6FN9B9McQ==" + "version": "16.11.9", + "resolved": "https://registry.npmjs.org/@types/node/-/node-16.11.9.tgz", + "integrity": "sha512-MKmdASMf3LtPzwLyRrFjtFFZ48cMf8jmX5VRYrDQiJa8Ybu5VAmkqBWqKU8fdCwD8ysw4mQ9nrEHvzg6gunR7A==" }, "@types/node-fetch": { "version": "2.5.12", @@ -702,9 +769,9 @@ } }, "acorn": { - "version": "8.4.1", - "resolved": "https://registry.npmjs.org/acorn/-/acorn-8.4.1.tgz", - "integrity": "sha512-asabaBSkEKosYKMITunzX177CXxQ4Q8BSSzMTKD+FefUhipQC70gfW5SiUDhYQ3vk8G+81HqQk7Fv9OXwwn9KA==" + "version": "8.6.0", + "resolved": "https://registry.npmjs.org/acorn/-/acorn-8.6.0.tgz", + "integrity": "sha512-U1riIR+lBSNi3IbxtaHOIKdH8sLFv3NYfNv8sg7ZsNhcfl4HF2++BfqqrNAxoCLQW1iiylOj76ecnaUxz+z9yw==" }, "acorn-globals": { "version": "6.0.0", @@ -768,9 +835,9 @@ } }, "adm-zip": { - "version": "0.5.7", - "resolved": "https://registry.npmjs.org/adm-zip/-/adm-zip-0.5.7.tgz", - "integrity": "sha512-QLEo3eoC2B0i3+g/G5nNzKbGoVOjW2ingZ4TXl7/YeDM+FAl3SiHSNnokTZLFEuVHBn5CbZ42KJcIIsRji1EgQ==" + "version": "0.5.9", + "resolved": "https://registry.npmjs.org/adm-zip/-/adm-zip-0.5.9.tgz", + "integrity": "sha512-s+3fXLkeeLjZ2kLjCBwQufpI5fuN+kIGBxu6530nVQZGVol0d7Y/M88/xw9HGGUcJjKf8LutN3VPRUBq6N7Ajg==" }, "after": { "version": "0.8.2", @@ -834,9 +901,9 @@ "dev": true }, "ansi-regex": { - "version": "2.1.1", - "resolved": "https://registry.npmjs.org/ansi-regex/-/ansi-regex-2.1.1.tgz", - "integrity": "sha1-w7M6te42DYbg5ijwRorn7yfWVN8=" + "version": "5.0.1", + "resolved": "https://registry.npmjs.org/ansi-regex/-/ansi-regex-5.0.1.tgz", + "integrity": "sha512-quJQXlTSUGL2LH9SUXo8VwsY4soanhgo6LNSm84E1LBcE8s3O0wpdiRzyR9z/ZZJMlMWv37qOOb9pdJlMUEKFQ==" }, "ansi-styles": { "version": "3.2.1", @@ -1197,9 +1264,9 @@ "integrity": "sha512-P8BjAsXvZS+VIDUI11hHCQEv74YT67YUi5JJFNWIqL235sBmjX4+qx9Muvls5ivyNENctx46xQLQ3aTuE7ssaQ==" }, "camelcase": { - "version": "6.2.0", - "resolved": "https://registry.npmjs.org/camelcase/-/camelcase-6.2.0.tgz", - "integrity": "sha512-c7wVvbw3f37nuobQNtgsgG9POC9qMbNuMQmTCqZv23b6MIz0fcYpBiOlv9gEN/hdLdnZTDQhg6e9Dq5M1vKvfg==", + "version": "6.2.1", + "resolved": "https://registry.npmjs.org/camelcase/-/camelcase-6.2.1.tgz", + "integrity": "sha512-tVI4q5jjFV5CavAU8DXfza/TJcZutVKo/5Foskmsqcm0MsL91moHvwiGNnqaa2o6PF/7yT5ikDRcVcl8Rj6LCA==", "dev": true }, "caseless": { @@ -1282,46 +1349,13 @@ "string-width": "^4.2.0", "strip-ansi": "^6.0.0", "wrap-ansi": "^7.0.0" - }, - "dependencies": { - "ansi-regex": { - "version": "5.0.0", - "resolved": "https://registry.npmjs.org/ansi-regex/-/ansi-regex-5.0.0.tgz", - "integrity": "sha512-bY6fj56OUQ0hU1KjFNDQuJFezqKdrAyFdIevADiqrWHwSlbmBNMHp5ak2f40Pm8JTFyM2mqxkG6ngkHO11f/lg==", - "dev": true - }, - "is-fullwidth-code-point": { - "version": "3.0.0", - "resolved": "https://registry.npmjs.org/is-fullwidth-code-point/-/is-fullwidth-code-point-3.0.0.tgz", - "integrity": "sha512-zymm5+u+sCsSWyD9qNaejV3DFvhCKclKdizYaJUuHA83RLjb7nSuGnddCHGv0hk+KY7BMAlsWeK4Ueg6EV6XQg==", - "dev": true - }, - "string-width": { - "version": "4.2.2", - "resolved": "https://registry.npmjs.org/string-width/-/string-width-4.2.2.tgz", - "integrity": "sha512-XBJbT3N4JhVumXE0eoLU9DCjcaF92KLNqTmFCnG1pf8duUxFGwtP6AD6nkjw9a3IdiRtL3E2w3JDiE/xi3vOeA==", - "dev": true, - "requires": { - "emoji-regex": "^8.0.0", - "is-fullwidth-code-point": "^3.0.0", - "strip-ansi": "^6.0.0" - } - }, - "strip-ansi": { - "version": "6.0.0", - "resolved": "https://registry.npmjs.org/strip-ansi/-/strip-ansi-6.0.0.tgz", - "integrity": "sha512-AuvKTrTfQNYNIctbR1K/YGTR1756GycPsg7b9bdV9Duqur4gv6aKqHXah67Z8ImS7WEz5QVcOtlfW2rZEugt6w==", - "dev": true, - "requires": { - "ansi-regex": "^5.0.0" - } - } } }, "code-point-at": { "version": "1.1.0", "resolved": "https://registry.npmjs.org/code-point-at/-/code-point-at-1.1.0.tgz", - "integrity": "sha1-DQcLTQQ6W+ozovGkDi7bPZpMz3c=" + "integrity": "sha1-DQcLTQQ6W+ozovGkDi7bPZpMz3c=", + "optional": true }, "color-convert": { "version": "1.9.3", @@ -1336,6 +1370,12 @@ "resolved": "https://registry.npmjs.org/color-name/-/color-name-1.1.3.tgz", "integrity": "sha1-p9BVi9icQveV3UIyj3QIMcpTvCU=" }, + "color-support": { + "version": "1.1.3", + "resolved": "https://registry.npmjs.org/color-support/-/color-support-1.1.3.tgz", + "integrity": "sha512-qiBjkpbMLO/HL68y+lh4q0/O1MZFj2RX6X/KmMa3+gJD3z+WwI1ZzDHysvqHGS3mP6mznPckpXmw1nI9cJjyRg==", + "optional": true + }, "combined-stream": { "version": "1.0.8", "resolved": "https://registry.npmjs.org/combined-stream/-/combined-stream-1.0.8.tgz", @@ -1399,12 +1439,19 @@ "integrity": "sha512-+Hp8fLp57wnUSt0tY0tHEXh4voZRDnoIrZPqlo3DPiI4y9lwg/jqx+1Om94/W6ZaPDOUbnjOt/99w66zk+l1Xg==" }, "cookie-parser": { - "version": "1.4.5", - "resolved": "https://registry.npmjs.org/cookie-parser/-/cookie-parser-1.4.5.tgz", - "integrity": "sha512-f13bPUj/gG/5mDr+xLmSxxDsB9DQiTIfhJS/sqjrmfAWiAN+x2O4i/XguTL9yDZ+/IFDanJ+5x7hC4CXT9Tdzw==", + "version": "1.4.6", + "resolved": "https://registry.npmjs.org/cookie-parser/-/cookie-parser-1.4.6.tgz", + "integrity": "sha512-z3IzaNjdwUC2olLIB5/ITd0/setiaFMLYiZJle7xg5Fe9KWAceil7xszYfHHBtDFYLSgJduS2Ty0P1uJdPDJeA==", "requires": { - "cookie": "0.4.0", + "cookie": "0.4.1", "cookie-signature": "1.0.6" + }, + "dependencies": { + "cookie": { + "version": "0.4.1", + "resolved": "https://registry.npmjs.org/cookie/-/cookie-0.4.1.tgz", + "integrity": "sha512-ZwrFkGJxUR3EIoXtO+yVE69Eb7KlixbaeAWfBQB9vVsNn/o+Yw69gBWSSDK825hQNdN+wF8zELf3dFNl/kxkUA==" + } } }, "cookie-signature": { @@ -1462,13 +1509,13 @@ } }, "data-urls": { - "version": "3.0.0", - "resolved": "https://registry.npmjs.org/data-urls/-/data-urls-3.0.0.tgz", - "integrity": "sha512-4AefxbTTdFtxDUdh0BuMBs2qJVL25Mow2zlcuuePegQwgD6GEmQao42LLEeksOui8nL4RcNEugIpFP7eRd33xg==", + "version": "3.0.1", + "resolved": "https://registry.npmjs.org/data-urls/-/data-urls-3.0.1.tgz", + "integrity": "sha512-Ds554NeT5Gennfoo9KN50Vh6tpgtvYEwraYjejXnyTpu1C7oXKxdFk75REooENHE8ndTVOJuv+BEs4/J/xcozw==", "requires": { "abab": "^2.0.3", - "whatwg-mimetype": "^2.3.0", - "whatwg-url": "^9.0.0" + "whatwg-mimetype": "^3.0.0", + "whatwg-url": "^10.0.0" } }, "date-utils": { @@ -1568,18 +1615,11 @@ } }, "domexception": { - "version": "2.0.1", - "resolved": "https://registry.npmjs.org/domexception/-/domexception-2.0.1.tgz", - "integrity": "sha512-yxJ2mFy/sibVQlu5qHjOkf9J3K6zgmCxgJ94u2EdvDOV09H+32LtRswEcUsmUWN72pVLOEnTSRaIVVzVQgS0dg==", + "version": "4.0.0", + "resolved": "https://registry.npmjs.org/domexception/-/domexception-4.0.0.tgz", + "integrity": "sha512-A2is4PLG+eeSfoTMA95/s4pvAoSo2mKtiM5jlHkAVewmiO8ISFTFKZjH7UAM1Atli/OT/7JHOrJRJiMKUZKYBw==", "requires": { - "webidl-conversions": "^5.0.0" - }, - "dependencies": { - "webidl-conversions": { - "version": "5.0.0", - "resolved": "https://registry.npmjs.org/webidl-conversions/-/webidl-conversions-5.0.0.tgz", - "integrity": "sha512-VlZwKPCkYKxQgeSbH5EyngOmRp7Ww7I9rQLERETtf5ofd9pGeswWiOtogpEO850jziPRarreGxn5QIiTqpb2wA==" - } + "webidl-conversions": "^7.0.0" } }, "ecc-jsbn": { @@ -1622,6 +1662,11 @@ "lodash": "^4.17.10" }, "dependencies": { + "ansi-regex": { + "version": "2.1.1", + "resolved": "https://registry.npmjs.org/ansi-regex/-/ansi-regex-2.1.1.tgz", + "integrity": "sha1-w7M6te42DYbg5ijwRorn7yfWVN8=" + }, "ansi-styles": { "version": "2.2.1", "resolved": "https://registry.npmjs.org/ansi-styles/-/ansi-styles-2.2.1.tgz", @@ -1639,6 +1684,14 @@ "supports-color": "^2.0.0" } }, + "strip-ansi": { + "version": "3.0.1", + "resolved": "https://registry.npmjs.org/strip-ansi/-/strip-ansi-3.0.1.tgz", + "integrity": "sha1-ajhfuIU9lS1f8F0Oiq+UJ43GPc8=", + "requires": { + "ansi-regex": "^2.0.0" + } + }, "supports-color": { "version": "2.0.0", "resolved": "https://registry.npmjs.org/supports-color/-/supports-color-2.0.0.tgz", @@ -1649,8 +1702,7 @@ "emoji-regex": { "version": "8.0.0", "resolved": "https://registry.npmjs.org/emoji-regex/-/emoji-regex-8.0.0.tgz", - "integrity": "sha512-MSjYzcWNOA0ewAHpz0MxpYFvwg6yjy1NG3xteoqz644VCo/RPgnr1/GGt+ic3iJTzQ8Eu3TdM14SawnVUmGE6A==", - "dev": true + "integrity": "sha512-MSjYzcWNOA0ewAHpz0MxpYFvwg6yjy1NG3xteoqz644VCo/RPgnr1/GGt+ic3iJTzQ8Eu3TdM14SawnVUmGE6A==" }, "encodeurl": { "version": "1.0.2", @@ -2323,9 +2375,9 @@ } }, "express-rate-limit": { - "version": "5.5.0", - "resolved": "https://registry.npmjs.org/express-rate-limit/-/express-rate-limit-5.5.0.tgz", - "integrity": "sha512-/1mrKggjXMxd1/ghPub5N3d36u5VlK8KjbQFQLxYub09BWSSgSXMQbXgFiIW0BYxjM49YCj8bkihONZR2U4+mQ==" + "version": "5.5.1", + "resolved": "https://registry.npmjs.org/express-rate-limit/-/express-rate-limit-5.5.1.tgz", + "integrity": "sha512-MTjE2eIbHv5DyfuFz4zLYWxpqVhEhkTiwFGuB74Q9CSou2WHO52nlE5y3Zlg6SIsiYUIPj6ifFxnkPz6O3sIUg==" }, "express-session": { "version": "1.17.2", @@ -2468,9 +2520,9 @@ "dev": true }, "follow-redirects": { - "version": "1.14.4", - "resolved": "https://registry.npmjs.org/follow-redirects/-/follow-redirects-1.14.4.tgz", - "integrity": "sha512-zwGkiSXC1MUJG/qmeIFH2HBJx9u0V46QGUe3YR1fXG8bXQxq7fLj0RjLZQ5nubr9qNJUZrH+xUcwXEoXNpfS+g==" + "version": "1.14.5", + "resolved": "https://registry.npmjs.org/follow-redirects/-/follow-redirects-1.14.5.tgz", + "integrity": "sha512-wtphSXy7d4/OR+MvIFbCVBDzZ5520qV8XfPklSN5QtxuMUJZ+b0Wnst1e1lCDocfzuCkHqj8k0FpZqO+UIaKNA==" }, "forever-agent": { "version": "0.6.1", @@ -2554,6 +2606,43 @@ "string-width": "^1.0.1", "strip-ansi": "^3.0.1", "wide-align": "^1.1.0" + }, + "dependencies": { + "ansi-regex": { + "version": "2.1.1", + "resolved": "https://registry.npmjs.org/ansi-regex/-/ansi-regex-2.1.1.tgz", + "integrity": "sha1-w7M6te42DYbg5ijwRorn7yfWVN8=", + "optional": true + }, + "is-fullwidth-code-point": { + "version": "1.0.0", + "resolved": "https://registry.npmjs.org/is-fullwidth-code-point/-/is-fullwidth-code-point-1.0.0.tgz", + "integrity": "sha1-754xOG8DGn8NZDr4L95QxFfvAMs=", + "optional": true, + "requires": { + "number-is-nan": "^1.0.0" + } + }, + "string-width": { + "version": "1.0.2", + "resolved": "https://registry.npmjs.org/string-width/-/string-width-1.0.2.tgz", + "integrity": "sha1-EYvfW4zcUaKn5w0hHgfisLmxB9M=", + "optional": true, + "requires": { + "code-point-at": "^1.0.0", + "is-fullwidth-code-point": "^1.0.0", + "strip-ansi": "^3.0.0" + } + }, + "strip-ansi": { + "version": "3.0.1", + "resolved": "https://registry.npmjs.org/strip-ansi/-/strip-ansi-3.0.1.tgz", + "integrity": "sha1-ajhfuIU9lS1f8F0Oiq+UJ43GPc8=", + "optional": true, + "requires": { + "ansi-regex": "^2.0.0" + } + } } }, "get-caller-file": { @@ -2675,6 +2764,13 @@ "integrity": "sha1-NPUEnOHs3ysGSa8+8k5F7TVBbZE=", "requires": { "ansi-regex": "^2.0.0" + }, + "dependencies": { + "ansi-regex": { + "version": "2.1.1", + "resolved": "https://registry.npmjs.org/ansi-regex/-/ansi-regex-2.1.1.tgz", + "integrity": "sha1-w7M6te42DYbg5ijwRorn7yfWVN8=" + } } }, "has-binary2": { @@ -2785,11 +2881,11 @@ "dev": true }, "html-encoding-sniffer": { - "version": "2.0.1", - "resolved": "https://registry.npmjs.org/html-encoding-sniffer/-/html-encoding-sniffer-2.0.1.tgz", - "integrity": "sha512-D5JbOMBIR/TVZkubHT+OyT2705QvogUW4IBn6nHd756OwieSF9aDYFj4dv6HHEVGYbHaLETa3WggZYWWMyy3ZQ==", + "version": "3.0.0", + "resolved": "https://registry.npmjs.org/html-encoding-sniffer/-/html-encoding-sniffer-3.0.0.tgz", + "integrity": "sha512-oWv4T4yJ52iKrufjnyZPkrN0CH3QnrUqdB6In1g5Fe1mia8GmF36gnfNySxoZtxD5+NmYw1EElVXiBk93UeskA==", "requires": { - "whatwg-encoding": "^1.0.5" + "whatwg-encoding": "^2.0.0" } }, "html-void-elements": { @@ -2798,15 +2894,15 @@ "integrity": "sha512-uE/TxKuyNIcx44cIWnjr/rfIATDH7ZaOMmstu0CwhFG1Dunhlp4OC6/NMbhiwoq5BpW0ubi303qnEk/PZj614w==" }, "http-errors": { - "version": "1.8.0", - "resolved": "https://registry.npmjs.org/http-errors/-/http-errors-1.8.0.tgz", - "integrity": "sha512-4I8r0C5JDhT5VkvI47QktDW75rNlGVsUf/8hzjCC/wkWI/jdTRmBb9aI7erSG82r1bjKY3F6k28WnsVxB1C73A==", + "version": "1.8.1", + "resolved": "https://registry.npmjs.org/http-errors/-/http-errors-1.8.1.tgz", + "integrity": "sha512-Kpk9Sm7NmI+RHhnj6OIWDI1d6fIoFAtFt9RLaTMRlg/8w49juAStsrBgp0Dp4OdxdVbRIeKhtCUvoi/RuAhO4g==", "requires": { "depd": "~1.1.2", "inherits": "2.0.4", "setprototypeof": "1.2.0", "statuses": ">= 1.5.0 < 2", - "toidentifier": "1.0.0" + "toidentifier": "1.0.1" }, "dependencies": { "inherits": { @@ -2818,6 +2914,11 @@ "version": "1.2.0", "resolved": "https://registry.npmjs.org/setprototypeof/-/setprototypeof-1.2.0.tgz", "integrity": "sha512-E5LDX7Wrp85Kil5bhZv46j8jOeboKq5JMmYM3gVGdGH8xFpPWXUMsNrlODCrkoxMEeNi/XZIwuRvY4XNwYMJpw==" + }, + "toidentifier": { + "version": "1.0.1", + "resolved": "https://registry.npmjs.org/toidentifier/-/toidentifier-1.0.1.tgz", + "integrity": "sha512-o5sSPKEkg/DIQNmH43V0/uerLrpzVedkUh8tGNvaeXpfpuwjKenlSox/2O/BTlZUtEe+JG7s5YhEz608PlAHRA==" } } }, @@ -2998,12 +3099,9 @@ "dev": true }, "is-fullwidth-code-point": { - "version": "1.0.0", - "resolved": "https://registry.npmjs.org/is-fullwidth-code-point/-/is-fullwidth-code-point-1.0.0.tgz", - "integrity": "sha1-754xOG8DGn8NZDr4L95QxFfvAMs=", - "requires": { - "number-is-nan": "^1.0.0" - } + "version": "3.0.0", + "resolved": "https://registry.npmjs.org/is-fullwidth-code-point/-/is-fullwidth-code-point-3.0.0.tgz", + "integrity": "sha512-zymm5+u+sCsSWyD9qNaejV3DFvhCKclKdizYaJUuHA83RLjb7nSuGnddCHGv0hk+KY7BMAlsWeK4Ueg6EV6XQg==" }, "is-glob": { "version": "4.0.1", @@ -3128,22 +3226,22 @@ "integrity": "sha1-peZUwuWi3rXyAdls77yoDA7y9RM=" }, "jsdom": { - "version": "17.0.0", - "resolved": "https://registry.npmjs.org/jsdom/-/jsdom-17.0.0.tgz", - "integrity": "sha512-MUq4XdqwtNurZDVeKScENMPHnkgmdIvMzZ1r1NSwHkDuaqI6BouPjr+17COo4/19oLNnmdpFDPOHVpgIZmZ+VA==", + "version": "18.1.0", + "resolved": "https://registry.npmjs.org/jsdom/-/jsdom-18.1.0.tgz", + "integrity": "sha512-q6QFAfSGLEUqRJ+GCV6vn6ItZCMARWh1d33wiJZPxc+wMNw7HK71JPmQ4C2lIZAsBH8TiJu4uplach/UcrC6bQ==", "requires": { "abab": "^2.0.5", - "acorn": "^8.4.1", + "acorn": "^8.5.0", "acorn-globals": "^6.0.0", "cssom": "^0.5.0", "cssstyle": "^2.3.0", - "data-urls": "^3.0.0", + "data-urls": "^3.0.1", "decimal.js": "^10.3.1", - "domexception": "^2.0.1", + "domexception": "^4.0.0", "escodegen": "^2.0.0", "form-data": "^4.0.0", - "html-encoding-sniffer": "^2.0.1", - "http-proxy-agent": "^4.0.1", + "html-encoding-sniffer": "^3.0.0", + "http-proxy-agent": "^5.0.0", "https-proxy-agent": "^5.0.0", "is-potential-custom-element-name": "^1.0.1", "nwsapi": "^2.2.0", @@ -3152,13 +3250,48 @@ "symbol-tree": "^3.2.4", "tough-cookie": "^4.0.0", "w3c-hr-time": "^1.0.2", - "w3c-xmlserializer": "^2.0.0", - "webidl-conversions": "^6.1.0", - "whatwg-encoding": "^1.0.5", - "whatwg-mimetype": "^2.3.0", - "whatwg-url": "^9.0.0", - "ws": "^8.0.0", - "xml-name-validator": "^3.0.0" + "w3c-xmlserializer": "^3.0.0", + "webidl-conversions": "^7.0.0", + "whatwg-encoding": "^2.0.0", + "whatwg-mimetype": "^3.0.0", + "whatwg-url": "^10.0.0", + "ws": "^8.2.3", + "xml-name-validator": "^4.0.0" + }, + "dependencies": { + "@tootallnate/once": { + "version": "2.0.0", + "resolved": "https://registry.npmjs.org/@tootallnate/once/-/once-2.0.0.tgz", + "integrity": "sha512-XCuKFP5PS55gnMVu3dty8KPatLqUoy/ZYzDzAGCQ8JNFCkLXzmI7vNHCR+XpbZaMWQK/vQubr7PkYq8g470J/A==" + }, + "debug": { + "version": "4.3.2", + "resolved": "https://registry.npmjs.org/debug/-/debug-4.3.2.tgz", + "integrity": "sha512-mOp8wKcvj7XxC78zLgw/ZA+6TSgkoE2C/ienthhRD298T7UNwAg9diBpLRxC0mOezLl4B0xV7M0cCO6P/O0Xhw==", + "requires": { + "ms": "2.1.2" + } + }, + "http-proxy-agent": { + "version": "5.0.0", + "resolved": "https://registry.npmjs.org/http-proxy-agent/-/http-proxy-agent-5.0.0.tgz", + "integrity": "sha512-n2hY8YdoRE1i7r6M0w9DIw5GgZN0G25P8zLCRQ8rjXtTU3vsNFBI/vWK/UIeE6g5MUUz6avwAPXmL6Fy9D/90w==", + "requires": { + "@tootallnate/once": "2", + "agent-base": "6", + "debug": "4" + } + }, + "ms": { + "version": "2.1.2", + "resolved": "https://registry.npmjs.org/ms/-/ms-2.1.2.tgz", + "integrity": "sha512-sGkPx+VjMtmA6MX27oA4FBFELFCZZ4S4XqeGOXCv68tT+jb3vk/RyaKWP0PTKyWtmLSM0b+adUTEvbs1PEaH2w==" + }, + "ws": { + "version": "8.2.3", + "resolved": "https://registry.npmjs.org/ws/-/ws-8.2.3.tgz", + "integrity": "sha512-wBuoj1BDpC6ZQ1B7DWQBYVLphPWkm8i9Y0/3YdHjHKHiohOJ1ws+3OccDWtH+PoC9DZD5WOTrJvNbWvjS6JWaA==" + } } }, "json-schema": { @@ -3582,16 +3715,16 @@ "integrity": "sha512-x0Vn8spI+wuJ1O6S7gnbaQg8Pxh4NNHb7KSINmEWKiPE4RKOplvijn+NkmYmmRgP68mc70j2EbeTFRsrswaQeg==" }, "mime-db": { - "version": "1.50.0", - "resolved": "https://registry.npmjs.org/mime-db/-/mime-db-1.50.0.tgz", - "integrity": "sha512-9tMZCDlYHqeERXEHO9f/hKfNXhre5dK2eE/krIvUjZbS2KPcqGDfNShIWS1uW9XOTKQKqK6qbeOci18rbfW77A==" + "version": "1.51.0", + "resolved": "https://registry.npmjs.org/mime-db/-/mime-db-1.51.0.tgz", + "integrity": "sha512-5y8A56jg7XVQx2mbv1lu49NR4dokRnhZYTtL+KGfaa27uq4pSTXkwQkFJl4pkRMyNFz/EtYDSkiiEHx3F7UN6g==" }, "mime-types": { - "version": "2.1.33", - "resolved": "https://registry.npmjs.org/mime-types/-/mime-types-2.1.33.tgz", - "integrity": "sha512-plLElXp7pRDd0bNZHw+nMd52vRYjLwQjygaNg7ddJ2uJtTlmnTCjWuPKxVu6//AdaRuME84SvLW91sIkBqGT0g==", + "version": "2.1.34", + "resolved": "https://registry.npmjs.org/mime-types/-/mime-types-2.1.34.tgz", + "integrity": "sha512-6cP692WwGIs9XXdOO4++N+7qjqv0rqxxVvJ3VHPh/Sc9mVZcQP+ZGhkKiTvWMQRr2tbHkJP/Yn7Y0npb3ZBs4A==", "requires": { - "mime-db": "1.50.0" + "mime-db": "1.51.0" } }, "mimic-response": { @@ -3646,16 +3779,16 @@ "optional": true }, "mocha": { - "version": "9.1.1", - "resolved": "https://registry.npmjs.org/mocha/-/mocha-9.1.1.tgz", - "integrity": "sha512-0wE74YMgOkCgBUj8VyIDwmLUjTsS13WV1Pg7l0SHea2qzZzlq7MDnfbPsHKcELBRk3+izEVkRofjmClpycudCA==", + "version": "9.1.3", + "resolved": "https://registry.npmjs.org/mocha/-/mocha-9.1.3.tgz", + "integrity": "sha512-Xcpl9FqXOAYqI3j79pEtHBBnQgVXIhpULjGQa7DVb0Po+VzmSIK9kanAiWLHoRR/dbZ2qpdPshuXr8l1VaHCzw==", "dev": true, "requires": { "@ungap/promise-all-settled": "1.1.2", "ansi-colors": "4.1.1", "browser-stdout": "1.3.1", "chokidar": "3.5.2", - "debug": "4.3.1", + "debug": "4.3.2", "diff": "5.0.0", "escape-string-regexp": "4.0.0", "find-up": "5.0.0", @@ -3666,12 +3799,11 @@ "log-symbols": "4.1.0", "minimatch": "3.0.4", "ms": "2.1.3", - "nanoid": "3.1.23", + "nanoid": "3.1.25", "serialize-javascript": "6.0.0", "strip-json-comments": "3.1.1", "supports-color": "8.1.1", "which": "2.0.2", - "wide-align": "1.1.3", "workerpool": "6.1.5", "yargs": "16.2.0", "yargs-parser": "20.2.4", @@ -3679,9 +3811,9 @@ }, "dependencies": { "debug": { - "version": "4.3.1", - "resolved": "https://registry.npmjs.org/debug/-/debug-4.3.1.tgz", - "integrity": "sha512-doEwdvm4PCeK4K3RQN2ZC2BYUBaxwLARCqZmMjtF8a51J2Rb0xpVloFRnCODwqjpwnAoao4pelN8l3RJdv3gRQ==", + "version": "4.3.2", + "resolved": "https://registry.npmjs.org/debug/-/debug-4.3.2.tgz", + "integrity": "sha512-mOp8wKcvj7XxC78zLgw/ZA+6TSgkoE2C/ienthhRD298T7UNwAg9diBpLRxC0mOezLl4B0xV7M0cCO6P/O0Xhw==", "dev": true, "requires": { "ms": "2.1.2" @@ -3713,12 +3845,6 @@ "integrity": "sha512-6FlzubTLZG3J2a/NVCAleEhjzq5oxgHyaCU9yYXvcLsvoVaHJq/s5xXI6/XXP6tz7R9xAOtHnSO/tXtF3WRTlA==", "dev": true }, - "strip-json-comments": { - "version": "3.1.1", - "resolved": "https://registry.npmjs.org/strip-json-comments/-/strip-json-comments-3.1.1.tgz", - "integrity": "sha512-6fPc+R4ihwqP6N/aIv2f1gMH8lOVtWQHoqC4yK6oSDVVocumAsfCqjkXnqiYMhmMwS/mEHLp7Vehlt3ql6lEig==", - "dev": true - }, "supports-color": { "version": "8.1.1", "resolved": "https://registry.npmjs.org/supports-color/-/supports-color-8.1.1.tgz", @@ -3745,9 +3871,9 @@ } }, "mongodb": { - "version": "3.7.2", - "resolved": "https://registry.npmjs.org/mongodb/-/mongodb-3.7.2.tgz", - "integrity": "sha512-/Qi0LmOjzIoV66Y2JQkqmIIfFOy7ZKsXnQNlUXPFXChOw3FCdNqVD5zvci9ybm6pkMe/Nw+Rz9I0Zsk2a+05iQ==", + "version": "3.7.3", + "resolved": "https://registry.npmjs.org/mongodb/-/mongodb-3.7.3.tgz", + "integrity": "sha512-Psm+g3/wHXhjBEktkxXsFMZvd3nemI0r3IPsE0bU+4//PnvNWKkzhZcEsbPcYiWqe8XqXJJEg4Tgtr7Raw67Yw==", "requires": { "bl": "^2.2.1", "bson": "^1.1.4", @@ -3763,9 +3889,9 @@ "integrity": "sha1-VgiurfwAvmwpAd9fmGF4jeDVl8g=" }, "msal": { - "version": "1.4.14", - "resolved": "https://registry.npmjs.org/msal/-/msal-1.4.14.tgz", - "integrity": "sha512-k8M5+/jbfSQoCf7CyQzBP5HE5mY8TkBujykLGTEp2x0MvOK/FQsfUTNis28zlvvPVzhgrhb5GQiGM8rRpXyHdA==", + "version": "1.4.15", + "resolved": "https://registry.npmjs.org/msal/-/msal-1.4.15.tgz", + "integrity": "sha512-H/CxkeZJ4laEK6GZ/cDKQoYjBTvDNFK3hDC8mfU8IkuZvKFfFdo9KM89r8spXY7xnBK9SQBAjIuQgwUogeUw7g==", "requires": { "tslib": "^1.9.3" }, @@ -3778,9 +3904,9 @@ } }, "mssql": { - "version": "7.2.1", - "resolved": "https://registry.npmjs.org/mssql/-/mssql-7.2.1.tgz", - "integrity": "sha512-kq0hVeD1tR+ikZqmLwgQqLGSavOhrrwaiYsYxdUQASifc3oIOFRx2IHpuWk+8oLI6Ab/s3o3JfpFX1v1Nf2sxA==", + "version": "7.3.0", + "resolved": "https://registry.npmjs.org/mssql/-/mssql-7.3.0.tgz", + "integrity": "sha512-3NGxDomH5Lci2g0EUrsejHIsvtFwlIE6A9SNFWQ2/JD4Dh0+5XVFHeyB4RXKb+nRMDosSUBAQDIVSuLXo5XFZA==", "requires": { "@tediousjs/connection-string": "^0.3.0", "debug": "^4.3.2", @@ -3867,9 +3993,9 @@ } }, "nanoid": { - "version": "3.1.23", - "resolved": "https://registry.npmjs.org/nanoid/-/nanoid-3.1.23.tgz", - "integrity": "sha512-FiB0kzdP0FFVGDKlRLEQ1BgDzU87dy5NnzjeW9YZNt+/c3+q82EQDUwniSAUxp/F0gFNI1ZhKU1FqYsMuqZVnw==", + "version": "3.1.25", + "resolved": "https://registry.npmjs.org/nanoid/-/nanoid-3.1.25.tgz", + "integrity": "sha512-rdwtIXaXCLFAQbnfqDRnI6jaRHp9fTcYBjtFKE8eezcZ7LuLjhUaQGNeMXf1HmRoCH32CLz6XwX0TtxEOS/A3Q==", "dev": true }, "napi-build-utils": { @@ -3907,6 +4033,15 @@ "path-to-regexp": "^1.7.0" }, "dependencies": { + "@sinonjs/fake-timers": { + "version": "7.1.2", + "resolved": "https://registry.npmjs.org/@sinonjs/fake-timers/-/fake-timers-7.1.2.tgz", + "integrity": "sha512-iQADsW4LBMISqZ6Ci1dupJL9pprqwcVFTcOsEmQOEhW+KLCVn/Y4Jrvg2k19fIHCp+iFprriYPTdRcQR8NbUPg==", + "dev": true, + "requires": { + "@sinonjs/commons": "^1.7.0" + } + }, "path-to-regexp": { "version": "1.8.0", "resolved": "https://registry.npmjs.org/path-to-regexp/-/path-to-regexp-1.8.0.tgz", @@ -3947,9 +4082,9 @@ "optional": true }, "node-fetch": { - "version": "2.6.5", - "resolved": "https://registry.npmjs.org/node-fetch/-/node-fetch-2.6.5.tgz", - "integrity": "sha512-mmlIVHJEu5rnIxgEgez6b9GgWXbkZj5YZ7fx+2r94a2E+Uirsp6HsPTPlomfdHtpt/B0cdKviwkoaM6pyvUOpQ==", + "version": "2.6.6", + "resolved": "https://registry.npmjs.org/node-fetch/-/node-fetch-2.6.6.tgz", + "integrity": "sha512-Z8/6vRlTUChSdIgMa51jxQ4lrw/Jy5SOW10ObaA47/RElsAN2c5Pn8bTgFGWn/ibwzXTE8qwr1Yzx28vsecXEA==", "requires": { "whatwg-url": "^5.0.0" }, @@ -7135,7 +7270,8 @@ "number-is-nan": { "version": "1.0.1", "resolved": "https://registry.npmjs.org/number-is-nan/-/number-is-nan-1.0.1.tgz", - "integrity": "sha1-CXtgK1NCKlIsGvuHkDGDNpQaAR0=" + "integrity": "sha1-CXtgK1NCKlIsGvuHkDGDNpQaAR0=", + "optional": true }, "nwsapi": { "version": "2.2.0", @@ -7194,12 +7330,12 @@ } }, "openapi-backend": { - "version": "4.2.0", - "resolved": "https://registry.npmjs.org/openapi-backend/-/openapi-backend-4.2.0.tgz", - "integrity": "sha512-eqdgJAjDbVZ7zhiIF68mlItFxqE48OPAM9nHHYx6BJMoGK2xInSBc2Oqp4dzsrsLIzoY8nVzK/vUtYktyXGb9Q==", + "version": "5.0.0", + "resolved": "https://registry.npmjs.org/openapi-backend/-/openapi-backend-5.0.0.tgz", + "integrity": "sha512-ppOFSLEXIgmRGcbd9kfPzXmUFHhlBux69rlu7dot5XZ6BH7ycXEvFjRdFLXZ76GdO++i3epDZkxkRBHqPNoz/Q==", "requires": { "@apidevtools/json-schema-ref-parser": "^9.0.7", - "ajv": "^8.5.0", + "ajv": "^8.6.2", "bath-es5": "^3.0.3", "cookie": "^0.4.0", "lodash": "^4.17.15", @@ -7231,20 +7367,20 @@ } }, "openapi-schema-validator": { - "version": "9.3.0", - "resolved": "https://registry.npmjs.org/openapi-schema-validator/-/openapi-schema-validator-9.3.0.tgz", - "integrity": "sha512-KlvgZMWTu+H1FHFSZNAGj369uXl3BD1nXSIq+sXlG6P+OrsAHd3YORx0ZEZ3WGdu2LQrPGmtowGQavYXL+PLwg==", + "version": "9.3.1", + "resolved": "https://registry.npmjs.org/openapi-schema-validator/-/openapi-schema-validator-9.3.1.tgz", + "integrity": "sha512-5wpFKMoEbUcjiqo16jIen3Cb2+oApSnYZpWn8WQdRO2q/dNQZZl8Pz6ESwCriiyU5AK4i5ZI6+7O3bHQr6+6+g==", "requires": { "ajv": "^8.1.0", "ajv-formats": "^2.0.2", "lodash.merge": "^4.6.1", - "openapi-types": "^9.3.0" + "openapi-types": "^9.3.1" } }, "openapi-types": { - "version": "9.3.0", - "resolved": "https://registry.npmjs.org/openapi-types/-/openapi-types-9.3.0.tgz", - "integrity": "sha512-sR23YjmuwDSMsQVZDHbV9mPgi0RyniQlqR0AQxTC2/F3cpSjRFMH3CFPjoWvNqhC4OxPkDYNb2l8Mc1Me6D/KQ==" + "version": "9.3.1", + "resolved": "https://registry.npmjs.org/openapi-types/-/openapi-types-9.3.1.tgz", + "integrity": "sha512-/Yvsd2D7miYB4HLJ3hOOS0+vnowQpaT75FsHzr/y5M9P4q9bwa7RcbW2YdH6KZBn8ceLbKGnHxMZ1CHliGHUFw==" }, "optional-js": { "version": "2.3.0", @@ -7568,9 +7704,9 @@ "integrity": "sha512-Hrgsx+orqoygnmhFbKaHE6c296J+HTAQXoxEF6gNupROmmGJRoyzfG3ccAveqCBrwr/2yxQ5BVd/GTl5agOwSg==" }, "rate-limiter-flexible": { - "version": "2.3.2", - "resolved": "https://registry.npmjs.org/rate-limiter-flexible/-/rate-limiter-flexible-2.3.2.tgz", - "integrity": "sha512-Q9isA+O+L5opvwB9sYAj49SYA0EA7fndVIKne0M9OSVpzaSZm3fv/9vE61B0c9A7PvLAxzeu0l/tYM2+JTi6qw==" + "version": "2.3.5", + "resolved": "https://registry.npmjs.org/rate-limiter-flexible/-/rate-limiter-flexible-2.3.5.tgz", + "integrity": "sha512-66QCGB8h74PklfrwDEFa8oIMHBL31x79WajtGnmS7LwJqdh8u/rnu4a8UNaxguB/YauTWdOI9lAM/WODVZw1FQ==" }, "raw-body": { "version": "2.4.0", @@ -7607,6 +7743,14 @@ "ini": "~1.3.0", "minimist": "^1.2.0", "strip-json-comments": "~2.0.1" + }, + "dependencies": { + "strip-json-comments": { + "version": "2.0.1", + "resolved": "https://registry.npmjs.org/strip-json-comments/-/strip-json-comments-2.0.1.tgz", + "integrity": "sha1-PFMZQukIwml8DsNEhYwobHygpgo=", + "optional": true + } } }, "readable-stream": { @@ -7845,26 +7989,15 @@ "integrity": "sha1-gRwwAxNoYTPvAAcSXjsO1wCXiBU=" }, "selenium-webdriver": { - "version": "4.0.0-rc-1", - "resolved": "https://registry.npmjs.org/selenium-webdriver/-/selenium-webdriver-4.0.0-rc-1.tgz", - "integrity": "sha512-bcrwFPRax8fifRP60p7xkWDGSJJoMkPAzufMlk5K2NyLPht/YZzR2WcIk1+3gR8VOCLlst1P2PI+MXACaFzpIw==", + "version": "4.0.0", + "resolved": "https://registry.npmjs.org/selenium-webdriver/-/selenium-webdriver-4.0.0.tgz", + "integrity": "sha512-tOlu6FnTjPq2FKpd153pl8o2cB7H40Rvl/ogiD2sapMv4IDjQqpIxbd+swDJe9UDLdszeh5CDis6lgy4e9UG1w==", "dev": true, "requires": { "jszip": "^3.6.0", "rimraf": "^3.0.2", "tmp": "^0.2.1", "ws": ">=7.4.6" - }, - "dependencies": { - "rimraf": { - "version": "3.0.2", - "resolved": "https://registry.npmjs.org/rimraf/-/rimraf-3.0.2.tgz", - "integrity": "sha512-JZkJMZkAGFFPP2YqXZXPbMlMBgsxzE8ILs4lMIX/2o0L9UBw9O/Y3o6wFw/i9YLapcUJWwqbi3kdxIPdC62TIA==", - "dev": true, - "requires": { - "glob": "^7.1.3" - } - } } }, "semver": { @@ -7986,9 +8119,9 @@ } }, "signal-exit": { - "version": "3.0.5", - "resolved": "https://registry.npmjs.org/signal-exit/-/signal-exit-3.0.5.tgz", - "integrity": "sha512-KWcOiKeQj6ZyXx7zq4YxSMgHRlod4czeBQZrPb8OKcohcqAXShm7E20kEMle9WBt26hFcAf0qLOcp5zmY7kOqQ==", + "version": "3.0.6", + "resolved": "https://registry.npmjs.org/signal-exit/-/signal-exit-3.0.6.tgz", + "integrity": "sha512-sDl4qMFpijcGw22U5w63KmD3cZJfBuFlVNbVMKje2keoKML7X2UzWbc4XrmEbDwg0NXJc3yv4/ox7b+JWb57kQ==", "optional": true }, "simple-concat": { @@ -8009,13 +8142,13 @@ } }, "simple-git": { - "version": "2.46.0", - "resolved": "https://registry.npmjs.org/simple-git/-/simple-git-2.46.0.tgz", - "integrity": "sha512-6eumII1vfP4NpRqxZcVWCcIT5xHH6dRyvBZSjkH4dJRDRpv+0f75hrN5ysp++y23Mfr3AbRC/dO2NDbfj1lJpQ==", + "version": "2.47.0", + "resolved": "https://registry.npmjs.org/simple-git/-/simple-git-2.47.0.tgz", + "integrity": "sha512-+HfCpqPBEZTPWiW9fPdbiPJDslM22MLqrktfzNKyI2pWaJa6DhfNVx4Mds04KZzVv5vjC9/ksw3y5gVf8ECWDg==", "requires": { "@kwsites/file-exists": "^1.1.1", "@kwsites/promise-deferred": "^1.1.1", - "debug": "^4.3.1" + "debug": "^4.3.2" }, "dependencies": { "debug": { @@ -8034,13 +8167,13 @@ } }, "sinon": { - "version": "11.1.2", - "resolved": "https://registry.npmjs.org/sinon/-/sinon-11.1.2.tgz", - "integrity": "sha512-59237HChms4kg7/sXhiRcUzdSkKuydDeTiamT/jesUVHshBgL8XAmhgFo0GfK6RruMDM/iRSij1EybmMog9cJw==", + "version": "12.0.1", + "resolved": "https://registry.npmjs.org/sinon/-/sinon-12.0.1.tgz", + "integrity": "sha512-iGu29Xhym33ydkAT+aNQFBINakjq69kKO6ByPvTsm3yyIACfyQttRTP03aBP/I8GfhFmLzrnKwNNkr0ORb1udg==", "dev": true, "requires": { "@sinonjs/commons": "^1.8.3", - "@sinonjs/fake-timers": "^7.1.2", + "@sinonjs/fake-timers": "^8.1.0", "@sinonjs/samsam": "^6.0.2", "diff": "^5.0.0", "nise": "^5.1.0", @@ -8224,9 +8357,9 @@ "integrity": "sha512-UjgapumWlbMhkBgzT7Ykc5YXUT46F0iKu8SGXq0bcwP5dz/h0Plj6enJqjz1Zbq2l5WaqYnrVbwWOWMyF3F47g==" }, "source-map-support": { - "version": "0.5.20", - "resolved": "https://registry.npmjs.org/source-map-support/-/source-map-support-0.5.20.tgz", - "integrity": "sha512-n1lZZ8Ve4ksRqizaBQgxXDgKwttHDhyfQjA6YZZn8+AroHbsIz+JjwxQDxbp+7y5OYCI8t1Yk7etjD9CRd2hIw==", + "version": "0.5.21", + "resolved": "https://registry.npmjs.org/source-map-support/-/source-map-support-0.5.21.tgz", + "integrity": "sha512-uBHU3L3czsIyYXKX88fdrGovxdSCoTGDRZ6SYXtSRxLZUzHg5P/66Ht6uoUlHu9EZod+inXhKo3qQgwXUT/y1w==", "requires": { "buffer-from": "^1.0.0", "source-map": "^0.6.0" @@ -8332,13 +8465,13 @@ "integrity": "sha512-KXDYZ9dszj6bzvnEMRYvxgeTHU74QBFL54XKtP3nyMuJ81CFYtABZ3bAzL2EdFUaEwJOBOgENyFj3R7oTzDyyw==" }, "string-width": { - "version": "1.0.2", - "resolved": "https://registry.npmjs.org/string-width/-/string-width-1.0.2.tgz", - "integrity": "sha1-EYvfW4zcUaKn5w0hHgfisLmxB9M=", + "version": "4.2.3", + "resolved": "https://registry.npmjs.org/string-width/-/string-width-4.2.3.tgz", + "integrity": "sha512-wKyQRQpjJ0sIp62ErSZdGsjMJWsap5oRNihHhu6G7JVO/9jIB6UyevL+tXuOqrng8j/cxKTWyWUwvSTriiZz/g==", "requires": { - "code-point-at": "^1.0.0", - "is-fullwidth-code-point": "^1.0.0", - "strip-ansi": "^3.0.0" + "emoji-regex": "^8.0.0", + "is-fullwidth-code-point": "^3.0.0", + "strip-ansi": "^6.0.1" } }, "string_decoder": { @@ -8357,18 +8490,18 @@ } }, "strip-ansi": { - "version": "3.0.1", - "resolved": "https://registry.npmjs.org/strip-ansi/-/strip-ansi-3.0.1.tgz", - "integrity": "sha1-ajhfuIU9lS1f8F0Oiq+UJ43GPc8=", + "version": "6.0.1", + "resolved": "https://registry.npmjs.org/strip-ansi/-/strip-ansi-6.0.1.tgz", + "integrity": "sha512-Y38VPSHcqkFrCpFnQ9vuSXmquuv5oXOKpGeT6aGrr3o3Gc9AlVa6JBfUSOCnbxGGZF+/0ooI7KrPuUSztUdU5A==", "requires": { - "ansi-regex": "^2.0.0" + "ansi-regex": "^5.0.1" } }, "strip-json-comments": { - "version": "2.0.1", - "resolved": "https://registry.npmjs.org/strip-json-comments/-/strip-json-comments-2.0.1.tgz", - "integrity": "sha1-PFMZQukIwml8DsNEhYwobHygpgo=", - "optional": true + "version": "3.1.1", + "resolved": "https://registry.npmjs.org/strip-json-comments/-/strip-json-comments-3.1.1.tgz", + "integrity": "sha512-6fPc+R4ihwqP6N/aIv2f1gMH8lOVtWQHoqC4yK6oSDVVocumAsfCqjkXnqiYMhmMwS/mEHLp7Vehlt3ql6lEig==", + "dev": true }, "superagent": { "version": "6.1.0", @@ -8716,9 +8849,9 @@ } }, "terser": { - "version": "5.9.0", - "resolved": "https://registry.npmjs.org/terser/-/terser-5.9.0.tgz", - "integrity": "sha512-h5hxa23sCdpzcye/7b8YqbE5OwKca/ni0RQz1uRX3tGh8haaGHqcuSqbGRybuAKNdntZ0mDgFNXPJ48xQ2RXKQ==", + "version": "5.10.0", + "resolved": "https://registry.npmjs.org/terser/-/terser-5.10.0.tgz", + "integrity": "sha512-AMmF99DMfEDiRJfxfY5jj5wNH/bYO09cniSqhfoyxc8sFoYIgkJy86G04UoZU5VjlpnplVu0K6Tx6E9b5+DlHA==", "requires": { "commander": "^2.20.0", "source-map": "~0.7.2", @@ -8785,17 +8918,6 @@ "dev": true, "requires": { "rimraf": "^3.0.0" - }, - "dependencies": { - "rimraf": { - "version": "3.0.2", - "resolved": "https://registry.npmjs.org/rimraf/-/rimraf-3.0.2.tgz", - "integrity": "sha512-JZkJMZkAGFFPP2YqXZXPbMlMBgsxzE8ILs4lMIX/2o0L9UBw9O/Y3o6wFw/i9YLapcUJWwqbi3kdxIPdC62TIA==", - "dev": true, - "requires": { - "glob": "^7.1.3" - } - } } }, "to-array": { @@ -8828,9 +8950,9 @@ } }, "tr46": { - "version": "2.1.0", - "resolved": "https://registry.npmjs.org/tr46/-/tr46-2.1.0.tgz", - "integrity": "sha512-15Ih7phfcdP5YxqiB+iDtLoaTz4Nd35+IiAv0kQ5FNKHzXgdWqPoTIqEDDJmXceQt4JZk6lVPT8lnDlPpGDppw==", + "version": "3.0.0", + "resolved": "https://registry.npmjs.org/tr46/-/tr46-3.0.0.tgz", + "integrity": "sha512-l7FvfAHlcmulp8kr+flpQZmVwtu7nfRV7NZujtN0OqES8EL4O4e0qqzL0DC5gAvx/ZC/9lk6rhcUwYvkBnBnYA==", "requires": { "punycode": "^2.1.1" } @@ -8893,22 +9015,22 @@ } }, "ueberdb2": { - "version": "1.4.18", - "resolved": "https://registry.npmjs.org/ueberdb2/-/ueberdb2-1.4.18.tgz", - "integrity": "sha512-u0Joo4FpNPw4PeTJTPe6GIZBFscZ8DbIFuD0cd60mMbkBpAh7l039hhOxoAGHuF0eRM9QEEqPpOunlOOJ1TTeg==", + "version": "1.4.19", + "resolved": "https://registry.npmjs.org/ueberdb2/-/ueberdb2-1.4.19.tgz", + "integrity": "sha512-9D4C9Lpb4fIHf7mdGOd24oYmrE/snEiz907rgWjVOTH/cN4O1kOuRW7VEaUatj9zDUdbne6W9wwypat/CYtZDQ==", "requires": { - "async": "^3.2.1", + "async": "^3.2.2", "cassandra-driver": "^4.6.3", "dirty": "^1.1.3", "elasticsearch": "^16.7.2", - "mongodb": "^3.7.1", - "mssql": "^7.2.1", + "mongodb": "^3.7.3", + "mssql": "^7.3.0", "mysql": "2.18.1", "nano": "^9.0.5", "pg": "^8.7.1", "redis": "^3.1.2", "rethinkdb": "^2.4.2", - "simple-git": "^2.45.1", + "simple-git": "^2.47.0", "sqlite3": "github:mapbox/node-sqlite3#593c9d498be2510d286349134537e3bf89401c4a" } }, @@ -9044,11 +9166,11 @@ } }, "w3c-xmlserializer": { - "version": "2.0.0", - "resolved": "https://registry.npmjs.org/w3c-xmlserializer/-/w3c-xmlserializer-2.0.0.tgz", - "integrity": "sha512-4tzD0mF8iSiMiNs30BiLO3EpfGLZUT2MSX/G+o7ZywDzliWQ3OPtTZ0PTC3B3ca1UAf4cJMHB+2Bf56EriJuRA==", + "version": "3.0.0", + "resolved": "https://registry.npmjs.org/w3c-xmlserializer/-/w3c-xmlserializer-3.0.0.tgz", + "integrity": "sha512-3WFqGEgSXIyGhOmAFtlicJNMjEps8b1MG31NCA0/vOF9+nKMUW1ckhi9cnNHmf88Rzw5V+dwIwsm2C7X8k9aQg==", "requires": { - "xml-name-validator": "^3.0.0" + "xml-name-validator": "^4.0.0" } }, "web-namespaces": { @@ -9057,30 +9179,40 @@ "integrity": "sha512-wYxSGajtmoP4WxfejAPIr4l0fVh+jeMXZb08wNc0tMg6xsfZXj3cECqIK0G7ZAqUq0PP8WlMDtaOGVBTAWztNw==" }, "webidl-conversions": { - "version": "6.1.0", - "resolved": "https://registry.npmjs.org/webidl-conversions/-/webidl-conversions-6.1.0.tgz", - "integrity": "sha512-qBIvFLGiBpLjfwmYAaHPXsn+ho5xZnGvyGvsarywGNc8VyQJUMHJ8OBKGGrPER0okBeMDaan4mNBlgBROxuI8w==" + "version": "7.0.0", + "resolved": "https://registry.npmjs.org/webidl-conversions/-/webidl-conversions-7.0.0.tgz", + "integrity": "sha512-VwddBukDzu71offAQR975unBIGqfKZpM+8ZX6ySk8nYhVoo5CYaZyzt3YBvYtRtO+aoGlqxPg/B87NGVZ/fu6g==" }, "whatwg-encoding": { - "version": "1.0.5", - "resolved": "https://registry.npmjs.org/whatwg-encoding/-/whatwg-encoding-1.0.5.tgz", - "integrity": "sha512-b5lim54JOPN9HtzvK9HFXvBma/rnfFeqsic0hSpjtDbVxR3dJKLc+KB4V6GgiGOvl7CY/KNh8rxSo9DKQrnUEw==", + "version": "2.0.0", + "resolved": "https://registry.npmjs.org/whatwg-encoding/-/whatwg-encoding-2.0.0.tgz", + "integrity": "sha512-p41ogyeMUrw3jWclHWTQg1k05DSVXPLcVxRTYsXUk+ZooOCZLcoYgPZ/HL/D/N+uQPOtcp1me1WhBEaX02mhWg==", "requires": { - "iconv-lite": "0.4.24" + "iconv-lite": "0.6.3" + }, + "dependencies": { + "iconv-lite": { + "version": "0.6.3", + "resolved": "https://registry.npmjs.org/iconv-lite/-/iconv-lite-0.6.3.tgz", + "integrity": "sha512-4fCk79wshMdzMp2rH06qWrJE4iolqLhCUH+OiuIgU++RB0+94NlDL81atO7GX55uUKueo0txHNtvEyI6D7WdMw==", + "requires": { + "safer-buffer": ">= 2.1.2 < 3.0.0" + } + } } }, "whatwg-mimetype": { - "version": "2.3.0", - "resolved": "https://registry.npmjs.org/whatwg-mimetype/-/whatwg-mimetype-2.3.0.tgz", - "integrity": "sha512-M4yMwr6mAnQz76TbJm914+gPpB/nCwvZbJU28cUD6dR004SAxDLOOSUaB1JDRqLtaOV/vi0IC5lEAGFgrjGv/g==" + "version": "3.0.0", + "resolved": "https://registry.npmjs.org/whatwg-mimetype/-/whatwg-mimetype-3.0.0.tgz", + "integrity": "sha512-nt+N2dzIutVRxARx1nghPKGv1xHikU7HKdfafKkLNLindmPU/ch3U31NOCGGA/dmPcmb1VlofO0vnKAcsm0o/Q==" }, "whatwg-url": { - "version": "9.1.0", - "resolved": "https://registry.npmjs.org/whatwg-url/-/whatwg-url-9.1.0.tgz", - "integrity": "sha512-CQ0UcrPHyomtlOCot1TL77WyMIm/bCwrJ2D6AOKGwEczU9EpyoqAokfqrf/MioU9kHcMsmJZcg1egXix2KYEsA==", + "version": "10.0.0", + "resolved": "https://registry.npmjs.org/whatwg-url/-/whatwg-url-10.0.0.tgz", + "integrity": "sha512-CLxxCmdUby142H5FZzn4D8ikO1cmypvXVQktsgosNy4a4BHrDHeciBBGZhb0bNoR5/MltoCatso+vFjjGx8t0w==", "requires": { - "tr46": "^2.1.0", - "webidl-conversions": "^6.1.0" + "tr46": "^3.0.0", + "webidl-conversions": "^7.0.0" } }, "which": { @@ -9092,11 +9224,12 @@ } }, "wide-align": { - "version": "1.1.3", - "resolved": "https://registry.npmjs.org/wide-align/-/wide-align-1.1.3.tgz", - "integrity": "sha512-QGkOQc8XL6Bt5PwnsExKBPuMKBxnGxWWW3fU55Xt4feHozMUhdUMaBCk290qpm/wG5u/RSKzwdAC4i51YigihA==", + "version": "1.1.5", + "resolved": "https://registry.npmjs.org/wide-align/-/wide-align-1.1.5.tgz", + "integrity": "sha512-eDMORYaPNZ4sQIuuYPDHdQvf4gyCF9rEEV/yPxGfwPkRodwEgiMUUXTx/dex+Me0wxx53S+NgUHaP7y3MGlDmg==", + "optional": true, "requires": { - "string-width": "^1.0.2 || 2" + "string-width": "^1.0.2 || 2 || 3 || 4" } }, "word-wrap": { @@ -9121,12 +9254,6 @@ "strip-ansi": "^6.0.0" }, "dependencies": { - "ansi-regex": { - "version": "5.0.0", - "resolved": "https://registry.npmjs.org/ansi-regex/-/ansi-regex-5.0.0.tgz", - "integrity": "sha512-bY6fj56OUQ0hU1KjFNDQuJFezqKdrAyFdIevADiqrWHwSlbmBNMHp5ak2f40Pm8JTFyM2mqxkG6ngkHO11f/lg==", - "dev": true - }, "ansi-styles": { "version": "4.3.0", "resolved": "https://registry.npmjs.org/ansi-styles/-/ansi-styles-4.3.0.tgz", @@ -9150,32 +9277,6 @@ "resolved": "https://registry.npmjs.org/color-name/-/color-name-1.1.4.tgz", "integrity": "sha512-dOy+3AuW3a2wNbZHIuMZpTcgjGuLU/uBL/ubcZF9OXbDo8ff4O8yVp5Bf0efS8uEoYo5q4Fx7dY9OgQGXgAsQA==", "dev": true - }, - "is-fullwidth-code-point": { - "version": "3.0.0", - "resolved": "https://registry.npmjs.org/is-fullwidth-code-point/-/is-fullwidth-code-point-3.0.0.tgz", - "integrity": "sha512-zymm5+u+sCsSWyD9qNaejV3DFvhCKclKdizYaJUuHA83RLjb7nSuGnddCHGv0hk+KY7BMAlsWeK4Ueg6EV6XQg==", - "dev": true - }, - "string-width": { - "version": "4.2.2", - "resolved": "https://registry.npmjs.org/string-width/-/string-width-4.2.2.tgz", - "integrity": "sha512-XBJbT3N4JhVumXE0eoLU9DCjcaF92KLNqTmFCnG1pf8duUxFGwtP6AD6nkjw9a3IdiRtL3E2w3JDiE/xi3vOeA==", - "dev": true, - "requires": { - "emoji-regex": "^8.0.0", - "is-fullwidth-code-point": "^3.0.0", - "strip-ansi": "^6.0.0" - } - }, - "strip-ansi": { - "version": "6.0.0", - "resolved": "https://registry.npmjs.org/strip-ansi/-/strip-ansi-6.0.0.tgz", - "integrity": "sha512-AuvKTrTfQNYNIctbR1K/YGTR1756GycPsg7b9bdV9Duqur4gv6aKqHXah67Z8ImS7WEz5QVcOtlfW2rZEugt6w==", - "dev": true, - "requires": { - "ansi-regex": "^5.0.0" - } } } }, @@ -9187,7 +9288,8 @@ "ws": { "version": "8.2.1", "resolved": "https://registry.npmjs.org/ws/-/ws-8.2.1.tgz", - "integrity": "sha512-XkgWpJU3sHU7gX8f13NqTn6KQ85bd1WU7noBHTT8fSohx7OS1TPY8k+cyRPCzFkia7C4mM229yeHr1qK9sM4JQ==" + "integrity": "sha512-XkgWpJU3sHU7gX8f13NqTn6KQ85bd1WU7noBHTT8fSohx7OS1TPY8k+cyRPCzFkia7C4mM229yeHr1qK9sM4JQ==", + "dev": true }, "wtfnode": { "version": "0.9.1", @@ -9195,9 +9297,9 @@ "integrity": "sha512-Ip6C2KeQPl/F3aP1EfOnPoQk14Udd9lffpoqWDNH3Xt78svxPbv53ngtmtfI0q2Te3oTq79XKTnRNXVIn/GsPA==" }, "xml-name-validator": { - "version": "3.0.0", - "resolved": "https://registry.npmjs.org/xml-name-validator/-/xml-name-validator-3.0.0.tgz", - "integrity": "sha512-A5CUptxDsvxKJEU3yO6DuWBSJz/qizqzJKOMIfUJHETbBw/sFaDxgd6fxm1ewUaM0jZ444Fc5vC5ROYurg/4Pw==" + "version": "4.0.0", + "resolved": "https://registry.npmjs.org/xml-name-validator/-/xml-name-validator-4.0.0.tgz", + "integrity": "sha512-ICP2e+jsHvAj2E2lIHxa5tjXRlKDJo4IdvPvCXbXQGdzSfmSpNVyIKMvoZHjDY9DP0zV17iI85o90vRFXNccRw==" }, "xml2js": { "version": "0.4.23", @@ -9257,40 +9359,6 @@ "string-width": "^4.2.0", "y18n": "^5.0.5", "yargs-parser": "^20.2.2" - }, - "dependencies": { - "ansi-regex": { - "version": "5.0.0", - "resolved": "https://registry.npmjs.org/ansi-regex/-/ansi-regex-5.0.0.tgz", - "integrity": "sha512-bY6fj56OUQ0hU1KjFNDQuJFezqKdrAyFdIevADiqrWHwSlbmBNMHp5ak2f40Pm8JTFyM2mqxkG6ngkHO11f/lg==", - "dev": true - }, - "is-fullwidth-code-point": { - "version": "3.0.0", - "resolved": "https://registry.npmjs.org/is-fullwidth-code-point/-/is-fullwidth-code-point-3.0.0.tgz", - "integrity": "sha512-zymm5+u+sCsSWyD9qNaejV3DFvhCKclKdizYaJUuHA83RLjb7nSuGnddCHGv0hk+KY7BMAlsWeK4Ueg6EV6XQg==", - "dev": true - }, - "string-width": { - "version": "4.2.2", - "resolved": "https://registry.npmjs.org/string-width/-/string-width-4.2.2.tgz", - "integrity": "sha512-XBJbT3N4JhVumXE0eoLU9DCjcaF92KLNqTmFCnG1pf8duUxFGwtP6AD6nkjw9a3IdiRtL3E2w3JDiE/xi3vOeA==", - "dev": true, - "requires": { - "emoji-regex": "^8.0.0", - "is-fullwidth-code-point": "^3.0.0", - "strip-ansi": "^6.0.0" - } - }, - "strip-ansi": { - "version": "6.0.0", - "resolved": "https://registry.npmjs.org/strip-ansi/-/strip-ansi-6.0.0.tgz", - "integrity": "sha512-AuvKTrTfQNYNIctbR1K/YGTR1756GycPsg7b9bdV9Duqur4gv6aKqHXah67Z8ImS7WEz5QVcOtlfW2rZEugt6w==", - "dev": true, - "requires": { - "ansi-regex": "^5.0.0" - } - } } }, "yargs-parser": { diff --git a/src/package.json b/src/package.json index 1c7dba7de..d369cdfb9 100644 --- a/src/package.json +++ b/src/package.json @@ -32,30 +32,30 @@ "dependencies": { "async": "^3.2.2", "clean-css": "^5.2.2", - "cookie-parser": "1.4.5", + "cookie-parser": "^1.4.6", "cross-spawn": "^7.0.3", "ejs": "^3.1.6", "etherpad-require-kernel": "^1.0.15", "etherpad-yajsml": "0.0.12", "express": "4.17.1", - "express-rate-limit": "5.5.0", + "express-rate-limit": "^5.5.1", "express-session": "1.17.2", "fast-deep-equal": "^3.1.3", "find-root": "1.1.0", - "formidable": "1.2.6", - "http-errors": "1.8.0", + "formidable": "^1.2.6", + "http-errors": "^1.8.1", "js-cookie": "^3.0.1", - "jsdom": "^17.0.0", + "jsdom": "^18.1.0", "jsonminify": "0.4.1", "languages4translatewiki": "0.1.3", "lodash.clonedeep": "4.5.0", "log4js": "0.6.38", "measured-core": "^2.0.0", - "mime-types": "^2.1.33", + "mime-types": "^2.1.34", "npm": "^6.14.15", - "openapi-backend": "^4.2.0", + "openapi-backend": "^5.0.0", "proxy-addr": "^2.0.7", - "rate-limiter-flexible": "^2.3.2", + "rate-limiter-flexible": "^2.3.5", "rehype": "^11.0.0", "rehype-minify-whitespace": "^4.0.5", "request": "2.88.2", @@ -63,11 +63,11 @@ "security": "1.0.0", "semver": "^7.3.5", "socket.io": "^2.4.1", - "terser": "^5.9.0", + "terser": "^5.10.0", "threads": "^1.7.0", "tiny-worker": "^2.3.0", "tinycon": "0.6.8", - "ueberdb2": "^1.4.18", + "ueberdb2": "^1.4.19", "underscore": "1.13.1", "unorm": "1.6.0", "wtfnode": "^0.9.1" @@ -86,13 +86,13 @@ "eslint-plugin-promise": "^5.1.1", "eslint-plugin-you-dont-need-lodash-underscore": "^6.12.0", "etherpad-cli-client": "^0.1.12", - "mocha": "^9.1.1", + "mocha": "^9.1.3", "mocha-froth": "^0.2.10", "nodeify": "^1.0.1", "openapi-schema-validation": "^0.4.2", - "selenium-webdriver": "^4.0.0-rc-1", + "selenium-webdriver": "^4.0.0", "set-cookie-parser": "^2.4.8", - "sinon": "^11.1.2", + "sinon": "^12.0.1", "split-grid": "^1.0.11", "superagent": "^6.1.0", "supertest": "^6.1.6" From 9cd59a84af6d4bf1dd159e51eac3870982ec502b Mon Sep 17 00:00:00 2001 From: Richard Hansen Date: Mon, 22 Nov 2021 17:25:00 -0500 Subject: [PATCH 019/446] Fix bug_report.md bug template --- .github/ISSUE_TEMPLATE/bug_report.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/.github/ISSUE_TEMPLATE/bug_report.md b/.github/ISSUE_TEMPLATE/bug_report.md index 4adcdb102..041f7c1a9 100644 --- a/.github/ISSUE_TEMPLATE/bug_report.md +++ b/.github/ISSUE_TEMPLATE/bug_report.md @@ -1,5 +1,3 @@ -IMPORTANT: Please disable plugins prior to posting a bug report. If you have a problem with a plugin please post on the plugin repository. Thanks! - --- name: Bug report about: Create a report to help us improve @@ -9,6 +7,8 @@ assignees: '' --- + + **Describe the bug** A clear and concise description of what the bug is. From d74dd235a458576da5416d66f8b21be0f171bac1 Mon Sep 17 00:00:00 2001 From: Richard Hansen Date: Mon, 25 Oct 2021 01:16:46 -0400 Subject: [PATCH 020/446] Changeset: Replace `appendATextToAssembler()` with a generator --- CHANGELOG.md | 2 ++ src/node/db/Pad.js | 2 +- src/static/js/Changeset.js | 28 +++++++++++++++++++++------- src/static/js/ace2_inner.js | 2 +- src/tests/frontend/specs/easysync.js | 2 +- 5 files changed, 26 insertions(+), 10 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 799b875e3..5e1f3f565 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -19,6 +19,8 @@ * `eachAttribNumber()` * `makeAttribsString()` * `opAttributeValue()` + * `appendATextToAssembler()`: Deprecated in favor of the new `opsFromAText()` + generator function. # 1.8.15 diff --git a/src/node/db/Pad.js b/src/node/db/Pad.js index 677e9c014..d412037df 100644 --- a/src/node/db/Pad.js +++ b/src/node/db/Pad.js @@ -499,7 +499,7 @@ Pad.prototype.copyPadWithoutHistory = async function (destinationID, force) { // based on Changeset.makeSplice const assem = Changeset.smartOpAssembler(); - Changeset.appendATextToAssembler(oldAText, assem); + for (const op of Changeset.opsFromAText(oldAText)) assem.append(op); assem.endDocument(); // although we have instantiated the newPad with '\n', an additional '\n' is diff --git a/src/static/js/Changeset.js b/src/static/js/Changeset.js index 24cc855a2..949002c7d 100644 --- a/src/static/js/Changeset.js +++ b/src/static/js/Changeset.js @@ -1718,17 +1718,18 @@ exports.copyAText = (atext1, atext2) => { }; /** - * Append the set of operations from atext to an assembler. + * Convert AText to a series of operations. * - * @param {AText} atext - - * @param assem - Assembler like SmartOpAssembler TODO add desc + * @param {AText} atext - The AText to convert. + * @yields {Op} + * @returns {Generator} */ -exports.appendATextToAssembler = (atext, assem) => { +exports.opsFromAText = function* (atext) { // intentionally skips last newline char of atext const iter = exports.opIterator(atext.attribs); let lastOp = null; while (iter.hasNext()) { - if (lastOp != null) assem.append(lastOp); + if (lastOp != null) yield lastOp; lastOp = iter.next(); } if (lastOp == null) return; @@ -1741,11 +1742,24 @@ exports.appendATextToAssembler = (atext, assem) => { const lastLineLength = atext.text.length - nextToLastNewlineEnd - 1; lastOp.lines--; lastOp.chars -= (lastLineLength + 1); - assem.append(lastOp); + yield copyOp(lastOp); lastOp.lines = 0; lastOp.chars = lastLineLength; } - if (lastOp.chars) assem.append(lastOp); + if (lastOp.chars) yield lastOp; +}; + +/** + * Append the set of operations from atext to an assembler. + * + * @deprecated Use `opsFromAText` instead. + * @param {AText} atext - + * @param assem - Assembler like SmartOpAssembler TODO add desc + */ +exports.appendATextToAssembler = (atext, assem) => { + padutils.warnWithStack( + 'Changeset.appendATextToAssembler() is deprecated; use Changeset.opsFromAText() instead'); + for (const op of exports.opsFromAText(atext)) assem.append(op); }; /** diff --git a/src/static/js/ace2_inner.js b/src/static/js/ace2_inner.js index 46c6ca579..3597959ac 100644 --- a/src/static/js/ace2_inner.js +++ b/src/static/js/ace2_inner.js @@ -530,7 +530,7 @@ function Ace2Inner(editorInfo, cssManagers) { o.chars = lastLineLength; o.lines = 0; assem.append(o); - Changeset.appendATextToAssembler(atext, assem); + for (const op of Changeset.opsFromAText(atext)) assem.append(op); const newLen = oldLen + assem.getLengthChange(); const changeset = Changeset.checkRep( Changeset.pack(oldLen, newLen, assem.toString(), atext.text.slice(0, -1))); diff --git a/src/tests/frontend/specs/easysync.js b/src/tests/frontend/specs/easysync.js index 121218407..79e81f41f 100644 --- a/src/tests/frontend/specs/easysync.js +++ b/src/tests/frontend/specs/easysync.js @@ -745,7 +745,7 @@ describe('easysync', function () { const testAppendATextToAssembler = (testId, atext, correctOps) => { it(`testAppendATextToAssembler#${testId}`, async function () { const assem = Changeset.smartOpAssembler(); - Changeset.appendATextToAssembler(atext, assem); + for (const op of Changeset.opsFromAText(atext)) assem.append(op); expect(assem.toString()).to.equal(correctOps); }); }; From ed78b56079803d97d136d6496b668e7e59fb538c Mon Sep 17 00:00:00 2001 From: Richard Hansen Date: Mon, 22 Nov 2021 15:34:03 -0500 Subject: [PATCH 021/446] tests: Refine `copyPadWithoutHistory` tests --- src/tests/backend/specs/api/pad.js | 55 ++++++++++++------------------ 1 file changed, 22 insertions(+), 33 deletions(-) diff --git a/src/tests/backend/specs/api/pad.js b/src/tests/backend/specs/api/pad.js index e91689851..ab56c8ca4 100644 --- a/src/tests/backend/specs/api/pad.js +++ b/src/tests/backend/specs/api/pad.js @@ -48,22 +48,16 @@ const ulSpaceHtml = '
  • one
    • const expectedSpaceHtml = '
    • one
    '; describe(__filename, function () { - before(async function () { agent = await common.init(); }); + before(async function () { + agent = await common.init(); + const res = await agent.get('/api/') + .expect(200) + .expect('Content-Type', /json/); + apiVersion = res.body.currentVersion; + assert(apiVersion); + }); describe('Sanity checks', function () { - it('can connect', async function () { - await agent.get('/api/') - .expect(200) - .expect('Content-Type', /json/); - }); - - it('finds the version tag', async function () { - const res = await agent.get('/api/') - .expect(200); - apiVersion = res.body.currentVersion; - assert(apiVersion); - }); - it('errors with invalid APIKey', async function () { // This is broken because Etherpad doesn't handle HTTP codes properly see #2343 // If your APIKey is password you deserve to fail all tests anyway @@ -523,37 +517,32 @@ describe(__filename, function () { assert.equal(receivedHtml, expectedHtml); }); - describe('when try copy a pad with a group that does not exist', function () { - const padId = makeid(); - const padWithNonExistentGroup = `notExistentGroup$${padId}`; - it('throws an error', async function () { - const res = await agent.get(`${endPoint('copyPadWithoutHistory')}` + - `&sourceID=${sourcePadId}` + - `&destinationID=${padWithNonExistentGroup}&force=true`) - .expect(200); - assert.equal(res.body.code, 1); - }); + it('copying to a non-existent group throws an error', async function () { + const padWithNonExistentGroup = `notExistentGroup$${newPad}`; + const res = await agent.get(`${endPoint('copyPadWithoutHistory')}` + + `&sourceID=${sourcePadId}` + + `&destinationID=${padWithNonExistentGroup}&force=true`) + .expect(200); + assert.equal(res.body.code, 1); }); - describe('when try copy a pad and destination pad already exist', function () { - const padIdExistent = makeid(); - - before(async function () { - await createNewPadWithHtml(padIdExistent, ulHtml); + describe('copying to an existing pad', function () { + beforeEach(async function () { + await createNewPadWithHtml(newPad, ulHtml); }); - it('force=false throws an error', async function () { + it('force=false fails', async function () { const res = await agent.get(`${endPoint('copyPadWithoutHistory')}` + `&sourceID=${sourcePadId}` + - `&destinationID=${padIdExistent}&force=false`) + `&destinationID=${newPad}&force=false`) .expect(200); assert.equal(res.body.code, 1); }); - it('force=true returns a successful response', async function () { + it('force=true succeeds', async function () { const res = await agent.get(`${endPoint('copyPadWithoutHistory')}` + `&sourceID=${sourcePadId}` + - `&destinationID=${padIdExistent}&force=true`) + `&destinationID=${newPad}&force=true`) .expect(200); assert.equal(res.body.code, 0); }); From dab881139d1fa26683fce51676e1f59b45deb02f Mon Sep 17 00:00:00 2001 From: Richard Hansen Date: Mon, 22 Nov 2021 15:16:00 -0500 Subject: [PATCH 022/446] Pad: Fix `copyPadWithoutHistory` apool corruption bug --- CHANGELOG.md | 4 +- src/node/db/Pad.js | 3 +- src/static/js/AttributePool.js | 21 ++++++++- src/tests/backend/specs/api/pad.js | 73 ++++++++++++++++++++++++++++++ 4 files changed, 96 insertions(+), 5 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 5e1f3f565..e11c6b691 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,6 +1,8 @@ # 1.9.0 (not yet released) -### Notable enhancements +### Notable enhancements and fixes + +* Fixed a potential attribute pool corruption bug with `copyPadWithoutHistory`. #### For plugin authors diff --git a/src/node/db/Pad.js b/src/node/db/Pad.js index d412037df..2aed5fd23 100644 --- a/src/node/db/Pad.js +++ b/src/node/db/Pad.js @@ -492,10 +492,9 @@ Pad.prototype.copyPadWithoutHistory = async function (destinationID, force) { // initialize the pad with a new line to avoid getting the defaultText const newPad = await padManager.getPad(destinationID, '\n'); + newPad.pool = sourcePad.pool.clone(); const oldAText = this.atext; - const newPool = newPad.pool; - newPool.fromJsonable(sourcePad.pool.toJsonable()); // copy that sourceId pool to the new pad // based on Changeset.makeSplice const assem = Changeset.smartOpAssembler(); diff --git a/src/static/js/AttributePool.js b/src/static/js/AttributePool.js index d8e587654..d46b207bb 100644 --- a/src/static/js/AttributePool.js +++ b/src/static/js/AttributePool.js @@ -91,6 +91,17 @@ class AttributePool { this.nextNum = 0; } + /** + * @returns {AttributePool} A deep copy of this attribute pool. + */ + clone() { + const c = new AttributePool(); + for (const [n, a] of Object.entries(this.numToAttrib)) c.numToAttrib[n] = [a[0], a[1]]; + Object.assign(c.attribToNum, this.attribToNum); + c.nextNum = this.nextNum; + return c; + } + /** * Add an attribute to the attribute set, or query for an existing attribute identifier. * @@ -164,7 +175,10 @@ class AttributePool { /** * @returns {Jsonable} An object that can be passed to `fromJsonable` to reconstruct this - * attribute pool. The returned object can be converted to JSON. + * attribute pool. The returned object can be converted to JSON. WARNING: The returned object + * has references to internal state (it is not a deep copy). Use the `clone()` method to copy + * a pool -- do NOT do `new AttributePool().fromJsonable(pool.toJsonable())` to copy because + * the resulting shared state will lead to pool corruption. */ toJsonable() { return { @@ -177,7 +191,10 @@ class AttributePool { * Replace the contents of this attribute pool with values from a previous call to `toJsonable`. * * @param {Jsonable} obj - Object returned by `toJsonable` containing the attributes and their - * identifiers. + * identifiers. WARNING: This function takes ownership of the object (it does not make a deep + * copy). Use the `clone()` method to copy a pool -- do NOT do + * `new AttributePool().fromJsonable(pool.toJsonable())` to copy because the resulting shared + * state will lead to pool corruption. */ fromJsonable(obj) { this.numToAttrib = obj.numToAttrib; diff --git a/src/tests/backend/specs/api/pad.js b/src/tests/backend/specs/api/pad.js index ab56c8ca4..7aab137b9 100644 --- a/src/tests/backend/specs/api/pad.js +++ b/src/tests/backend/specs/api/pad.js @@ -9,6 +9,7 @@ const assert = require('assert').strict; const common = require('../../common'); +const padManager = require('../../../../node/db/PadManager'); let agent; const apiKey = common.apiKey; @@ -547,6 +548,78 @@ describe(__filename, function () { assert.equal(res.body.code, 0); }); }); + + // Regression test for https://github.com/ether/etherpad-lite/issues/5296 + it('source and destination attribute pools are independent', async function () { + // Strategy for this test: + // 1. Create a new pad without bold or italic text + // 2. Use copyPadWithoutHistory to copy the pad. + // 3. Add some bold text (but no italic text!) to the source pad. This should add a bold + // attribute to the source pad's pool but not to the destination pad's pool. + // 4. Add some italic text (but no bold text!) to the destination pad. This should add an + // italic attribute to the destination pad's pool with the same number as the newly added + // bold attribute in the source pad's pool. + // 5. Add some more text (bold or plain) to the source pad. This will save the source pad to + // the database after the destination pad has had an opportunity to corrupt the source + // pad. + // 6. Export the source and destination pads. Make sure that doesn't appear in the + // source pad's HTML, and that doesn't appear int he destination pad's HTML. + // 7. Force the server to re-init the pads from the database. + // 8. Repeat step 6. + // If appears in the source pad, or appears in the destination pad, then shared + // state between the two attribute pools caused corruption. + + const getHtml = async (padId) => { + const res = await agent.get(`${endPoint('getHTML')}&padID=${padId}`) + .expect(200) + .expect('Content-Type', /json/); + assert.equal(res.body.code, 0); + return res.body.data.html; + }; + + const setBody = async (padId, bodyHtml) => { + await agent.post(endPoint('setHTML')) + .send({padID: padId, html: `${bodyHtml}`}) + .expect(200) + .expect('Content-Type', /json/) + .expect((res) => assert.equal(res.body.code, 0)); + }; + + const origHtml = await getHtml(sourcePadId); + assert.doesNotMatch(origHtml, //); + assert.doesNotMatch(origHtml, //); + await agent.get(`${endPoint('copyPadWithoutHistory')}&sourceID=${sourcePadId}` + + `&destinationID=${newPad}&force=false`) + .expect(200) + .expect('Content-Type', /json/) + .expect((res) => assert.equal(res.body.code, 0)); + + const newBodySrc = 'bold'; + const newBodyDst = 'italic'; + await setBody(sourcePadId, newBodySrc); + await setBody(newPad, newBodyDst); + await setBody(sourcePadId, `${newBodySrc} foo`); + + let [srcHtml, dstHtml] = await Promise.all([getHtml(sourcePadId), getHtml(newPad)]); + assert.match(srcHtml, new RegExp(newBodySrc)); + assert.match(dstHtml, new RegExp(newBodyDst)); + + // Force the server to re-read the pads from the database. This rebuilds the attribute pool + // objects from scratch, ensuring that an internally inconsistent attribute pool object did + // not cause the above tests to accidentally pass. + const reInitPad = async (padId) => { + const pad = await padManager.getPad(padId); + await pad.init(); + }; + await Promise.all([ + reInitPad(sourcePadId), + reInitPad(newPad), + ]); + + [srcHtml, dstHtml] = await Promise.all([getHtml(sourcePadId), getHtml(newPad)]); + assert.match(srcHtml, new RegExp(newBodySrc)); + assert.match(dstHtml, new RegExp(newBodyDst)); + }); }); }); From fba0bb6dff9fe7e0bee904c551a0c7ea586df075 Mon Sep 17 00:00:00 2001 From: Richard Hansen Date: Sun, 21 Mar 2021 14:34:02 -0400 Subject: [PATCH 023/446] Changeset: Turn `textLinesMutator()` into a real class --- src/static/js/Changeset.js | 298 +++++++++++++-------------- src/tests/frontend/specs/easysync.js | 8 +- 2 files changed, 143 insertions(+), 163 deletions(-) diff --git a/src/static/js/Changeset.js b/src/static/js/Changeset.js index 949002c7d..fd34f39e4 100644 --- a/src/static/js/Changeset.js +++ b/src/static/js/Changeset.js @@ -639,43 +639,34 @@ exports.stringAssembler = () => ({ * actually a newline, but for the purposes of N and L values, the caller should pretend it is, and * for things to work right in that case, the input to the `insert` method should be a single line * with no newlines. - * - * @typedef {object} TextLinesMutator - * @property {Function} close - - * @property {Function} hasMore - - * @property {Function} insert - - * @property {Function} remove - - * @property {Function} removeLines - - * @property {Function} skip - - * @property {Function} skipLines - */ - -/** - * @param {(string[]|StringArrayLike)} lines - Lines to mutate (in place). - * @returns {TextLinesMutator} - */ -const textLinesMutator = (lines) => { +class TextLinesMutator { /** - * curSplice holds values that will be passed as arguments to lines.splice() to insert, delete, or - * change lines: - * - curSplice[0] is an index into the lines array. - * - curSplice[1] is the number of lines that will be removed from the lines array starting at - * the index. - * - The other elements represent mutated (changed by ops) lines or new lines (added by ops) to - * insert at the index. - * - * @type {[number, number?, ...string[]?]} + * @param {(string[]|StringArrayLike)} lines - Lines to mutate (in place). */ - const curSplice = [0, 0]; - let inSplice = false; - - // position in lines after curSplice is applied: - let curLine = 0; - let curCol = 0; - // invariant: if (inSplice) then (curLine is in curSplice[0] + curSplice.length - {2,3}) && - // curLine >= curSplice[0] - // invariant: if (inSplice && (curLine >= curSplice[0] + curSplice.length - 2)) then - // curCol == 0 + constructor(lines) { + this._lines = lines; + /** + * this._curSplice holds values that will be passed as arguments to this._lines.splice() to + * insert, delete, or change lines: + * - this._curSplice[0] is an index into the this._lines array. + * - this._curSplice[1] is the number of lines that will be removed from the this._lines array + * starting at the index. + * - The other elements represent mutated (changed by ops) lines or new lines (added by ops) + * to insert at the index. + * + * @type {[number, number?, ...string[]?]} + */ + this._curSplice = [0, 0]; + this._inSplice = false; + // position in lines after curSplice is applied: + this._curLine = 0; + this._curCol = 0; + // invariant: if (inSplice) then (curLine is in curSplice[0] + curSplice.length - {2,3}) && + // curLine >= curSplice[0] + // invariant: if (inSplice && (curLine >= curSplice[0] + curSplice.length - 2)) then + // curCol == 0 + } /** * Get a line from `lines` at given index. @@ -683,13 +674,13 @@ const textLinesMutator = (lines) => { * @param {number} idx - an index * @returns {string} */ - const linesGet = (idx) => { - if ('get' in lines) { - return lines.get(idx); + _linesGet(idx) { + if ('get' in this._lines) { + return this._lines.get(idx); } else { - return lines[idx]; + return this._lines[idx]; } - }; + } /** * Return a slice from `lines`. @@ -698,51 +689,50 @@ const textLinesMutator = (lines) => { * @param {number} end - the end index * @returns {string[]} */ - const linesSlice = (start, end) => { - if (lines.slice) { - return lines.slice(start, end); + _linesSlice(start, end) { + // can be unimplemented if removeLines's return value not needed + if (this._lines.slice) { + return this._lines.slice(start, end); } else { return []; } - }; + } /** * Return the length of `lines`. * * @returns {number} */ - const linesLength = () => { - if ((typeof lines.length) === 'number') { - return lines.length; + _linesLength() { + if (typeof this._lines.length === 'number') { + return this._lines.length; } else { - return lines.length(); + return this._lines.length(); } - }; + } /** * Starts a new splice. */ - const enterSplice = () => { - curSplice[0] = curLine; - curSplice[1] = 0; + _enterSplice() { + this._curSplice[0] = this._curLine; + this._curSplice[1] = 0; // TODO(doc) when is this the case? // check all enterSplice calls and changes to curCol - if (curCol > 0) { - putCurLineInSplice(); - } - inSplice = true; - }; + if (this._curCol > 0) this._putCurLineInSplice(); + this._inSplice = true; + } /** * Changes the lines array according to the values in curSplice and resets curSplice. Called via * close or TODO(doc). */ - const leaveSplice = () => { - lines.splice(...curSplice); - curSplice.length = 2; - curSplice[0] = curSplice[1] = 0; - inSplice = false; - }; + _leaveSplice() { + this._lines.splice(...this._curSplice); + this._curSplice.length = 2; + this._curSplice[0] = this._curSplice[1] = 0; + this._inSplice = false; + } /** * Indicates if curLine is already in the splice. This is necessary because the last element in @@ -752,20 +742,23 @@ const textLinesMutator = (lines) => { * * @returns {boolean} true if curLine is in splice */ - const isCurLineInSplice = () => (curLine - curSplice[0] < (curSplice.length - 2)); + _isCurLineInSplice() { + return this._curLine - this._curSplice[0] < this._curSplice.length - 2; + } /** * Incorporates current line into the splice and marks its old position to be deleted. * * @returns {number} the index of the added line in curSplice */ - const putCurLineInSplice = () => { - if (!isCurLineInSplice()) { - curSplice.push(linesGet(curSplice[0] + curSplice[1])); - curSplice[1]++; + _putCurLineInSplice() { + if (!this._isCurLineInSplice()) { + this._curSplice.push(this._linesGet(this._curSplice[0] + this._curSplice[1])); + this._curSplice[1]++; } - return 2 + curLine - curSplice[0]; // TODO should be the same as curSplice.length - 1 - }; + // TODO should be the same as this._curSplice.length - 1 + return 2 + this._curLine - this._curSplice[0]; + } /** * It will skip some newlines by putting them into the splice. @@ -773,30 +766,30 @@ const textLinesMutator = (lines) => { * @param {number} L - * @param {boolean} includeInSplice - indicates if attributes are present */ - const skipLines = (L, includeInSplice) => { + skipLines(L, includeInSplice) { if (!L) return; if (includeInSplice) { - if (!inSplice) enterSplice(); + if (!this._inSplice) this._enterSplice(); // TODO(doc) should this count the number of characters that are skipped to check? for (let i = 0; i < L; i++) { - curCol = 0; - putCurLineInSplice(); - curLine++; + this._curCol = 0; + this._putCurLineInSplice(); + this._curLine++; } } else { - if (inSplice) { + if (this._inSplice) { if (L > 1) { // TODO(doc) figure out why single lines are incorporated into splice instead of ignored - leaveSplice(); + this._leaveSplice(); } else { - putCurLineInSplice(); + this._putCurLineInSplice(); } } - curLine += L; - curCol = 0; + this._curLine += L; + this._curCol = 0; } // tests case foo in remove(), which isn't otherwise covered in current impl - }; + } /** * Skip some characters. Can contain newlines. @@ -805,20 +798,20 @@ const textLinesMutator = (lines) => { * @param {number} L - number of newlines to skip * @param {boolean} includeInSplice - indicates if attributes are present */ - const skip = (N, L, includeInSplice) => { + skip(N, L, includeInSplice) { if (!N) return; if (L) { - skipLines(L, includeInSplice); + this.skipLines(L, includeInSplice); } else { - if (includeInSplice && !inSplice) enterSplice(); - if (inSplice) { + if (includeInSplice && !this._inSplice) this._enterSplice(); + if (this._inSplice) { // although the line is put into splice curLine is not increased, because // only some chars are skipped, not the whole line - putCurLineInSplice(); + this._putCurLineInSplice(); } - curCol += N; + this._curCol += N; } - }; + } /** * Remove whole lines from lines array. @@ -826,9 +819,9 @@ const textLinesMutator = (lines) => { * @param {number} L - number of lines to remove * @returns {string} */ - const removeLines = (L) => { + removeLines(L) { if (!L) return ''; - if (!inSplice) enterSplice(); + if (!this._inSplice) this._enterSplice(); /** * Gets a string of joined lines after the end of the splice. @@ -837,32 +830,32 @@ const textLinesMutator = (lines) => { * @returns {string} joined lines */ const nextKLinesText = (k) => { - const m = curSplice[0] + curSplice[1]; - return linesSlice(m, m + k).join(''); + const m = this._curSplice[0] + this._curSplice[1]; + return this._linesSlice(m, m + k).join(''); }; let removed = ''; - if (isCurLineInSplice()) { - if (curCol === 0) { - removed = curSplice[curSplice.length - 1]; - curSplice.length--; + if (this._isCurLineInSplice()) { + if (this._curCol === 0) { + removed = this._curSplice[this._curSplice.length - 1]; + this._curSplice.length--; removed += nextKLinesText(L - 1); - curSplice[1] += L - 1; + this._curSplice[1] += L - 1; } else { removed = nextKLinesText(L - 1); - curSplice[1] += L - 1; - const sline = curSplice.length - 1; - removed = curSplice[sline].substring(curCol) + removed; - curSplice[sline] = curSplice[sline].substring(0, curCol) + - linesGet(curSplice[0] + curSplice[1]); - curSplice[1] += 1; + this._curSplice[1] += L - 1; + const sline = this._curSplice.length - 1; + removed = this._curSplice[sline].substring(this._curCol) + removed; + this._curSplice[sline] = this._curSplice[sline].substring(0, this._curCol) + + this._linesGet(this._curSplice[0] + this._curSplice[1]); + this._curSplice[1] += 1; } } else { removed = nextKLinesText(L); - curSplice[1] += L; + this._curSplice[1] += L; } return removed; - }; + } /** * Remove text from lines array. @@ -871,18 +864,18 @@ const textLinesMutator = (lines) => { * @param {number} L - lines to delete * @returns {string} */ - const remove = (N, L) => { + remove(N, L) { if (!N) return ''; - if (L) return removeLines(L); - if (!inSplice) enterSplice(); + if (L) return this.removeLines(L); + if (!this._inSplice) this._enterSplice(); // although the line is put into splice, curLine is not increased, because // only some chars are removed not the whole line - const sline = putCurLineInSplice(); - const removed = curSplice[sline].substring(curCol, curCol + N); - curSplice[sline] = curSplice[sline].substring(0, curCol) + - curSplice[sline].substring(curCol + N); + const sline = this._putCurLineInSplice(); + const removed = this._curSplice[sline].substring(this._curCol, this._curCol + N); + this._curSplice[sline] = this._curSplice[sline].substring(0, this._curCol) + + this._curSplice[sline].substring(this._curCol + N); return removed; - }; + } /** * Inserts text into lines array. @@ -890,82 +883,69 @@ const textLinesMutator = (lines) => { * @param {string} text - the text to insert * @param {number} L - number of newlines in text */ - const insert = (text, L) => { + insert(text, L) { if (!text) return; - if (!inSplice) enterSplice(); + if (!this._inSplice) this._enterSplice(); if (L) { const newLines = exports.splitTextLines(text); - if (isCurLineInSplice()) { - const sline = curSplice.length - 1; + if (this._isCurLineInSplice()) { + const sline = this._curSplice.length - 1; /** @type {string} */ - const theLine = curSplice[sline]; - const lineCol = curCol; + const theLine = this._curSplice[sline]; + const lineCol = this._curCol; // insert the first new line - curSplice[sline] = theLine.substring(0, lineCol) + newLines[0]; - curLine++; + this._curSplice[sline] = theLine.substring(0, lineCol) + newLines[0]; + this._curLine++; newLines.splice(0, 1); // insert the remaining new lines - curSplice.push(...newLines); - curLine += newLines.length; + this._curSplice.push(...newLines); + this._curLine += newLines.length; // insert the remaining chars from the "old" line (e.g. the line we were in // when we started to insert new lines) - curSplice.push(theLine.substring(lineCol)); - curCol = 0; // TODO(doc) why is this not set to the length of last line? + this._curSplice.push(theLine.substring(lineCol)); + this._curCol = 0; // TODO(doc) why is this not set to the length of last line? } else { - curSplice.push(...newLines); - curLine += newLines.length; + this._curSplice.push(...newLines); + this._curLine += newLines.length; } } else { // there are no additional lines // although the line is put into splice, curLine is not increased, because // there may be more chars in the line (newline is not reached) - const sline = putCurLineInSplice(); - if (!curSplice[sline]) { + const sline = this._putCurLineInSplice(); + if (!this._curSplice[sline]) { const err = new Error( 'curSplice[sline] not populated, actual curSplice contents is ' + - `${JSON.stringify(curSplice)}. Possibly related to ` + + `${JSON.stringify(this._curSplice)}. Possibly related to ` + 'https://github.com/ether/etherpad-lite/issues/2802'); console.error(err.stack || err.toString()); } - curSplice[sline] = curSplice[sline].substring(0, curCol) + text + - curSplice[sline].substring(curCol); - curCol += text.length; + this._curSplice[sline] = this._curSplice[sline].substring(0, this._curCol) + text + + this._curSplice[sline].substring(this._curCol); + this._curCol += text.length; } - }; + } /** * Checks if curLine (the line we are in when curSplice is applied) is the last line in `lines`. * * @returns {boolean} indicates if there are lines left */ - const hasMore = () => { - let docLines = linesLength(); - if (inSplice) { - docLines += curSplice.length - 2 - curSplice[1]; + hasMore() { + let docLines = this._linesLength(); + if (this._inSplice) { + docLines += this._curSplice.length - 2 - this._curSplice[1]; } - return curLine < docLines; - }; + return this._curLine < docLines; + } /** * Closes the splice */ - const close = () => { - if (inSplice) { - leaveSplice(); - } - }; - - const self = { - skip, - remove, - insert, - close, - hasMore, - removeLines, - skipLines, - }; - return self; -}; + close() { + if (this._inSplice) this._leaveSplice(); + } +} /** * Apply operations to other operations. @@ -1106,7 +1086,7 @@ exports.mutateTextLines = (cs, lines) => { const unpacked = exports.unpack(cs); const csIter = exports.opIterator(unpacked.ops); const bankIter = exports.stringIterator(unpacked.charBank); - const mut = textLinesMutator(lines); + const mut = new TextLinesMutator(lines); while (csIter.hasNext()) { const op = csIter.next(); switch (op.opcode) { @@ -1239,7 +1219,7 @@ exports.mutateAttributionLines = (cs, lines, pool) => { const csBank = unpacked.charBank; let csBankIndex = 0; // treat the attribution lines as text lines, mutating a line at a time - const mut = textLinesMutator(lines); + const mut = new TextLinesMutator(lines); /** @type {?OpIter} */ let lineIter = null; @@ -2310,7 +2290,7 @@ const followAttributes = (att1, att2, pool) => { }; exports.exportedForTestingOnly = { + TextLinesMutator, followAttributes, - textLinesMutator, toSplices, }; diff --git a/src/tests/frontend/specs/easysync.js b/src/tests/frontend/specs/easysync.js index 79e81f41f..3ffd2a211 100644 --- a/src/tests/frontend/specs/easysync.js +++ b/src/tests/frontend/specs/easysync.js @@ -92,7 +92,7 @@ describe('easysync', function () { const runMutationTest = (testId, origLines, muts, correct) => { it(`runMutationTest#${testId}`, async function () { let lines = origLines.slice(); - const mu = Changeset.exportedForTestingOnly.textLinesMutator(lines); + const mu = new Changeset.exportedForTestingOnly.TextLinesMutator(lines); applyMutations(mu, muts); mu.close(); expect(lines).to.eql(correct); @@ -211,7 +211,7 @@ describe('easysync', function () { const lines = ['1\n', '2\n', '3\n', '4\n']; let mu; - mu = Changeset.exportedForTestingOnly.textLinesMutator(lines); + mu = new Changeset.exportedForTestingOnly.TextLinesMutator(lines); expect(mu.hasMore()).to.be(true); mu.skip(8, 4); expect(mu.hasMore()).to.be(false); @@ -219,7 +219,7 @@ describe('easysync', function () { expect(mu.hasMore()).to.be(false); // still 1,2,3,4 - mu = Changeset.exportedForTestingOnly.textLinesMutator(lines); + mu = new Changeset.exportedForTestingOnly.TextLinesMutator(lines); expect(mu.hasMore()).to.be(true); mu.remove(2, 1); expect(mu.hasMore()).to.be(true); @@ -235,7 +235,7 @@ describe('easysync', function () { expect(mu.hasMore()).to.be(false); // 2,3,4,5 now - mu = Changeset.exportedForTestingOnly.textLinesMutator(lines); + mu = new Changeset.exportedForTestingOnly.TextLinesMutator(lines); expect(mu.hasMore()).to.be(true); mu.remove(6, 3); expect(mu.hasMore()).to.be(true); From 657492e191af4fc39f6cda6a995f8dffebfe6a2d Mon Sep 17 00:00:00 2001 From: Richard Hansen Date: Mon, 25 Oct 2021 01:21:19 -0400 Subject: [PATCH 024/446] Changeset: Turn `newOp()` into a real class --- CHANGELOG.md | 1 + src/node/utils/padDiff.js | 4 +- src/static/js/Changeset.js | 143 +++++++++++++++++---------- src/static/js/ace2_inner.js | 2 +- src/static/js/contentcollector.js | 2 +- src/static/js/linestylefilter.js | 2 +- src/tests/frontend/specs/easysync.js | 6 +- 7 files changed, 99 insertions(+), 61 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index e11c6b691..03aab8890 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -23,6 +23,7 @@ * `opAttributeValue()` * `appendATextToAssembler()`: Deprecated in favor of the new `opsFromAText()` generator function. + * `newOp()`: Deprecated in favor of the new `Op` class. # 1.8.15 diff --git a/src/node/utils/padDiff.js b/src/node/utils/padDiff.js index 05cba308f..d0c61633f 100644 --- a/src/node/utils/padDiff.js +++ b/src/node/utils/padDiff.js @@ -270,7 +270,7 @@ PadDiff.prototype._createDeletionChangeset = function (cs, startAText, apool) { let curChar = 0; let curLineOpIter = null; let curLineOpIterLine; - let curLineNextOp = Changeset.newOp('+'); + let curLineNextOp = new Changeset.Op('+'); const unpacked = Changeset.unpack(cs); const csIter = Changeset.opIterator(unpacked.ops); @@ -302,7 +302,7 @@ PadDiff.prototype._createDeletionChangeset = function (cs, startAText, apool) { } if (!curLineNextOp.chars) { - curLineNextOp = curLineOpIter.hasNext() ? curLineOpIter.next() : Changeset.newOp(); + curLineNextOp = curLineOpIter.hasNext() ? curLineOpIter.next() : new Changeset.Op(); } const charsToUse = Math.min(numChars, curLineNextOp.chars); diff --git a/src/static/js/Changeset.js b/src/static/js/Changeset.js index fd34f39e4..0b3b01a7c 100644 --- a/src/static/js/Changeset.js +++ b/src/static/js/Changeset.js @@ -83,31 +83,74 @@ exports.numToString = (num) => num.toString(36).toLowerCase(); /** * An operation to apply to a shared document. - * - * @typedef {object} Op - * @property {('+'|'-'|'='|'')} opcode - The operation's operator: - * - '=': Keep the next `chars` characters (containing `lines` newlines) from the base - * document. - * - '-': Remove the next `chars` characters (containing `lines` newlines) from the base - * document. - * - '+': Insert `chars` characters (containing `lines` newlines) at the current position in - * the document. The inserted characters come from the changeset's character bank. - * - '' (empty string): Invalid operator used in some contexts to signifiy the lack of an - * operation. - * @property {number} chars - The number of characters to keep, insert, or delete. - * @property {number} lines - The number of characters among the `chars` characters that are - * newlines. If non-zero, the last character must be a newline. - * @property {string} attribs - Identifiers of attributes to apply to the text, represented as a - * repeated (zero or more) sequence of asterisk followed by a non-negative base-36 (lower-case) - * integer. For example, '*2*1o' indicates that attributes 2 and 60 apply to the text affected - * by the operation. The identifiers come from the document's attribute pool. This is the empty - * string for remove ('-') operations. For keep ('=') operations, the attributes are merged with - * the base text's existing attributes: - * - A keep op attribute with a non-empty value replaces an existing base text attribute that - * has the same key. - * - A keep op attribute with an empty value is interpreted as an instruction to remove an - * existing base text attribute that has the same key, if one exists. */ +class Op { + /** + * @param {(''|'='|'+'|'-')} [opcode=''] - Initial value of the `opcode` property. + */ + constructor(opcode = '') { + /** + * The operation's operator: + * - '=': Keep the next `chars` characters (containing `lines` newlines) from the base + * document. + * - '-': Remove the next `chars` characters (containing `lines` newlines) from the base + * document. + * - '+': Insert `chars` characters (containing `lines` newlines) at the current position in + * the document. The inserted characters come from the changeset's character bank. + * - '' (empty string): Invalid operator used in some contexts to signifiy the lack of an + * operation. + * + * @type {(''|'='|'+'|'-')} + * @public + */ + this.opcode = opcode; + + /** + * The number of characters to keep, insert, or delete. + * + * @type {number} + * @public + */ + this.chars = 0; + + /** + * The number of characters among the `chars` characters that are newlines. If non-zero, the + * last character must be a newline. + * + * @type {number} + * @public + */ + this.lines = 0; + + /** + * Identifiers of attributes to apply to the text, represented as a repeated (zero or more) + * sequence of asterisk followed by a non-negative base-36 (lower-case) integer. For example, + * '*2*1o' indicates that attributes 2 and 60 apply to the text affected by the operation. The + * identifiers come from the document's attribute pool. + * + * For keep ('=') operations, the attributes are merged with the base text's existing + * attributes: + * - A keep op attribute with a non-empty value replaces an existing base text attribute that + * has the same key. + * - A keep op attribute with an empty value is interpreted as an instruction to remove an + * existing base text attribute that has the same key, if one exists. + * + * This is the empty string for remove ('-') operations. + * + * @type {string} + * @public + */ + this.attribs = ''; + } + + toString() { + if (!this.opcode) throw new TypeError('null op'); + if (typeof this.attribs !== 'string') throw new TypeError('attribs must be a string'); + const l = this.lines ? `|${exports.numToString(this.lines)}` : ''; + return this.attribs + l + this.opcode + exports.numToString(this.chars); + } +} +exports.Op = Op; /** * Describes changes to apply to a document. Does not include the attribute pool or the original @@ -166,8 +209,7 @@ exports.opIterator = (opsStr) => { }; let regexResult = nextRegexMatch(); - const next = (optOp) => { - const op = optOp || exports.newOp(); + const next = (op = new Op()) => { if (regexResult[0]) { op.attribs = regexResult[1]; op.lines = exports.parseNum(regexResult[2] || '0'); @@ -203,15 +245,14 @@ const clearOp = (op) => { /** * Creates a new Op object * + * @deprecated Use the `Op` class instead. * @param {('+'|'-'|'='|'')} [optOpcode=''] - The operation's operator. * @returns {Op} */ -exports.newOp = (optOpcode) => ({ - opcode: (optOpcode || ''), - chars: 0, - lines: 0, - attribs: '', -}); +exports.newOp = (optOpcode) => { + padutils.warnWithStack('Changeset.newOp() is deprecated; use the Changeset.Op class instead'); + return new Op(optOpcode); +}; /** * Copies op1 to op2 @@ -220,7 +261,7 @@ exports.newOp = (optOpcode) => ({ * @param {Op} [op2] - dest Op. If not given, a new Op is used. * @returns {Op} `op2` */ -const copyOp = (op1, op2 = exports.newOp()) => Object.assign(op2, op1); +const copyOp = (op1, op2 = new Op()) => Object.assign(op2, op1); /** * Serializes a sequence of Ops. @@ -257,7 +298,7 @@ const copyOp = (op1, op2 = exports.newOp()) => Object.assign(op2, op1); * @returns {Generator} */ const opsFromText = function* (opcode, text, attribs = '', pool = null) { - const op = exports.newOp(opcode); + const op = new Op(opcode); op.attribs = typeof attribs === 'string' ? attribs : new AttributeMap(pool).update(attribs || [], opcode === '+').toString(); const lastNewlinePos = text.lastIndexOf('\n'); @@ -447,7 +488,7 @@ exports.smartOpAssembler = () => { */ exports.mergingOpAssembler = () => { const assem = exports.opAssembler(); - const bufOp = exports.newOp(); + const bufOp = new Op(); // If we get, for example, insertions [xxx\n,yyy], those don't merge, // but if we get [xxx\n,yyy,zzz\n], that merges to [xxx\nyyyzzz\n]. @@ -523,12 +564,8 @@ exports.opAssembler = () => { * @param {Op} op - Operation to add. Ownership remains with the caller. */ const append = (op) => { - if (!op.opcode) throw new TypeError('null op'); - if (typeof op.attribs !== 'string') throw new TypeError('attribs must be a string'); - serialized += op.attribs; - if (op.lines) serialized += `|${exports.numToString(op.lines)}`; - serialized += op.opcode; - serialized += exports.numToString(op.chars); + assert(op instanceof Op, 'argument must be an instance of Op'); + serialized += op.toString(); }; const toString = () => serialized; @@ -972,8 +1009,8 @@ const applyZip = (in1, in2, func) => { const iter1 = exports.opIterator(in1); const iter2 = exports.opIterator(in2); const assem = exports.smartOpAssembler(); - const op1 = exports.newOp(); - const op2 = exports.newOp(); + const op1 = new Op(); + const op2 = new Op(); while (op1.opcode || iter1.hasNext() || op2.opcode || iter2.hasNext()) { if ((!op1.opcode) && iter1.hasNext()) iter1.next(op1); if ((!op2.opcode) && iter2.hasNext()) iter2.next(op2); @@ -1148,7 +1185,7 @@ exports.composeAttributes = (att1, att2, resultIsMutation, pool) => { * @returns {Op} The result of applying `csOp` to `attOp`. */ const slicerZipperFunc = (attOp, csOp, pool) => { - const opOut = exports.newOp(); + const opOut = new Op(); if (!attOp.opcode) { copyOp(csOp, opOut); csOp.opcode = ''; @@ -1231,7 +1268,7 @@ exports.mutateAttributionLines = (cs, lines, pool) => { const line = mut.removeLines(1); lineIter = exports.opIterator(line); } - if (!lineIter || !lineIter.hasNext()) return exports.newOp(); + if (!lineIter || !lineIter.hasNext()) return new Op(); return lineIter.next(); }; let lineAssem = null; @@ -1248,8 +1285,8 @@ exports.mutateAttributionLines = (cs, lines, pool) => { lineAssem = null; }; - let csOp = exports.newOp(); - let attOp = exports.newOp(); + let csOp = new Op(); + let attOp = new Op(); while (csOp.opcode || csIter.hasNext() || attOp.opcode || isNextMutOp()) { if (!csOp.opcode && csIter.hasNext()) csOp = csIter.next(); if ((!csOp.opcode) && (!attOp.opcode) && (!lineAssem) && (!(lineIter && lineIter.hasNext()))) { @@ -1826,7 +1863,7 @@ exports.attribsAttributeValue = (attribs, key, pool) => { */ exports.builder = (oldLen) => { const assem = exports.smartOpAssembler(); - const o = exports.newOp(); + const o = new Op(); const charBank = exports.stringAssembler(); const self = { @@ -1930,8 +1967,8 @@ exports.makeAttribsString = (opcode, attribs, pool) => { exports.subattribution = (astr, start, optEnd) => { const iter = exports.opIterator(astr); const assem = exports.smartOpAssembler(); - let attOp = exports.newOp(); - const csOp = exports.newOp(); + let attOp = new Op(); + const csOp = new Op(); const doCsOp = () => { if (!csOp.chars) return; @@ -1994,7 +2031,7 @@ exports.inverse = (cs, lines, alines, pool) => { let curChar = 0; let curLineOpIter = null; let curLineOpIterLine; - let curLineNextOp = exports.newOp('+'); + let curLineNextOp = new Op('+'); const unpacked = exports.unpack(cs); const csIter = exports.opIterator(unpacked.ops); @@ -2025,7 +2062,7 @@ exports.inverse = (cs, lines, alines, pool) => { curLineOpIter = exports.opIterator(alinesGet(curLine)); } if (!curLineNextOp.chars) { - curLineNextOp = curLineOpIter.hasNext() ? curLineOpIter.next() : exports.newOp(); + curLineNextOp = curLineOpIter.hasNext() ? curLineOpIter.next() : new Op(); } const charsToUse = Math.min(numChars, curLineNextOp.chars); func(charsToUse, curLineNextOp.attribs, charsToUse === curLineNextOp.chars && @@ -2135,7 +2172,7 @@ exports.follow = (cs1, cs2, reverseInsertOrder, pool) => { const hasInsertFirst = exports.attributeTester(['insertorder', 'first'], pool); const newOps = applyZip(unpacked1.ops, unpacked2.ops, (op1, op2) => { - const opOut = exports.newOp(); + const opOut = new Op(); if (op1.opcode === '+' || op2.opcode === '+') { let whichToDo; if (op2.opcode !== '+') { diff --git a/src/static/js/ace2_inner.js b/src/static/js/ace2_inner.js index 3597959ac..484205c06 100644 --- a/src/static/js/ace2_inner.js +++ b/src/static/js/ace2_inner.js @@ -523,7 +523,7 @@ function Ace2Inner(editorInfo, cssManagers) { const upToLastLine = rep.lines.offsetOfIndex(numLines - 1); const lastLineLength = rep.lines.atIndex(numLines - 1).text.length; const assem = Changeset.smartOpAssembler(); - const o = Changeset.newOp('-'); + const o = new Changeset.Op('-'); o.chars = upToLastLine; o.lines = numLines - 1; assem.append(o); diff --git a/src/static/js/contentcollector.js b/src/static/js/contentcollector.js index 2e6005bd8..7dd70e512 100644 --- a/src/static/js/contentcollector.js +++ b/src/static/js/contentcollector.js @@ -83,7 +83,7 @@ const makeContentCollector = (collectStyles, abrowser, apool, className2Author) const textArray = []; const attribsArray = []; let attribsBuilder = null; - const op = Changeset.newOp('+'); + const op = new Changeset.Op('+'); const self = { length: () => textArray.length, atColumnZero: () => textArray[textArray.length - 1] === '', diff --git a/src/static/js/linestylefilter.js b/src/static/js/linestylefilter.js index f6bcbb54e..19751999c 100644 --- a/src/static/js/linestylefilter.js +++ b/src/static/js/linestylefilter.js @@ -102,7 +102,7 @@ linestylefilter.getLineStyleFilter = (lineLength, aline, textAndClassFunc, apool let nextOp, nextOpClasses; const goNextOp = () => { - nextOp = attributionIter.hasNext() ? attributionIter.next() : Changeset.newOp(); + nextOp = attributionIter.hasNext() ? attributionIter.next() : new Changeset.Op(); nextOpClasses = (nextOp.opcode && attribsToClasses(nextOp.attribs)); }; goNextOp(); diff --git a/src/tests/frontend/specs/easysync.js b/src/tests/frontend/specs/easysync.js index 3ffd2a211..fc56c62a1 100644 --- a/src/tests/frontend/specs/easysync.js +++ b/src/tests/frontend/specs/easysync.js @@ -57,7 +57,7 @@ describe('easysync', function () { const mutationsToChangeset = (oldLen, arrayOfArrays) => { const assem = Changeset.smartOpAssembler(); - const op = Changeset.newOp(); + const op = new Changeset.Op(); const bank = Changeset.stringAssembler(); let oldPos = 0; let newLen = 0; @@ -507,7 +507,7 @@ describe('easysync', function () { const opAssem = Changeset.smartOpAssembler(); const oldLen = origText.length; - const nextOp = Changeset.newOp(); + const nextOp = new Changeset.Op(); const appendMultilineOp = (opcode, txt) => { nextOp.opcode = opcode; @@ -651,7 +651,7 @@ describe('easysync', function () { const testSplitJoinAttributionLines = (randomSeed) => { const stringToOps = (str) => { const assem = Changeset.mergingOpAssembler(); - const o = Changeset.newOp('+'); + const o = new Changeset.Op('+'); o.chars = 1; for (let i = 0; i < str.length; i++) { const c = str.charAt(i); From 86959f7ebc25665c26c7b3dd28184bd9fccc99e2 Mon Sep 17 00:00:00 2001 From: Richard Hansen Date: Thu, 21 Oct 2021 03:17:28 -0400 Subject: [PATCH 025/446] Changeset: Throw on unexpected chars while iterating ops --- src/static/js/Changeset.js | 15 +++++++-------- 1 file changed, 7 insertions(+), 8 deletions(-) diff --git a/src/static/js/Changeset.js b/src/static/js/Changeset.js index 0b3b01a7c..5d4871c5e 100644 --- a/src/static/js/Changeset.js +++ b/src/static/js/Changeset.js @@ -197,20 +197,21 @@ exports.newLen = (cs) => exports.unpack(cs).newLen; * @returns {OpIter} Operator iterator object. */ exports.opIterator = (opsStr) => { - const regex = /((?:\*[0-9a-z]+)*)(?:\|([0-9a-z]+))?([-+=])([0-9a-z]+)|\?|/g; + const regex = /((?:\*[0-9a-z]+)*)(?:\|([0-9a-z]+))?([-+=])([0-9a-z]+)|(.)/g; const nextRegexMatch = () => { const result = regex.exec(opsStr); - if (result[0] === '?') { - error('Hit error opcode in op stream'); - } - + if (!result) return null; + if (result[5] === '$') return null; // Start of the insert operation character bank. + if (result[5] != null) error(`invalid operation: ${opsStr.slice(regex.lastIndex - 1)}`); return result; }; let regexResult = nextRegexMatch(); + const hasNext = () => regexResult && !!regexResult[0]; + const next = (op = new Op()) => { - if (regexResult[0]) { + if (hasNext()) { op.attribs = regexResult[1]; op.lines = exports.parseNum(regexResult[2] || '0'); op.opcode = regexResult[3]; @@ -222,8 +223,6 @@ exports.opIterator = (opsStr) => { return op; }; - const hasNext = () => !!(regexResult[0]); - return { next, hasNext, From a4aec006dc332e245d8bd980c68e19ddc57a14ea Mon Sep 17 00:00:00 2001 From: Richard Hansen Date: Wed, 13 Oct 2021 15:42:03 -0400 Subject: [PATCH 026/446] Changeset: Turn `opIterator()` into a real class --- src/static/js/Changeset.js | 114 ++++++++++++++++++++----------------- 1 file changed, 63 insertions(+), 51 deletions(-) diff --git a/src/static/js/Changeset.js b/src/static/js/Changeset.js index 5d4871c5e..2b0e82482 100644 --- a/src/static/js/Changeset.js +++ b/src/static/js/Changeset.js @@ -184,11 +184,54 @@ exports.newLen = (cs) => exports.unpack(cs).newLen; * Iterator over a changeset's operations. * * Note: This class does NOT implement the ECMAScript iterable or iterator protocols. - * - * @typedef {object} OpIter - * @property {Function} hasNext - - * @property {Function} next - */ +class OpIter { + /** + * @param {string} ops - String encoding the change operations to iterate over. + */ + constructor(ops) { + this._ops = ops; + this._regex = /((?:\*[0-9a-z]+)*)(?:\|([0-9a-z]+))?([-+=])([0-9a-z]+)|(.)/g; + this._nextMatch = this._nextRegexMatch(); + } + + _nextRegexMatch() { + const match = this._regex.exec(this._ops); + if (!match) return null; + if (match[5] === '$') return null; // Start of the insert operation character bank. + if (match[5] != null) error(`invalid operation: ${this._ops.slice(this._regex.lastIndex - 1)}`); + return match; + } + + /** + * @returns {boolean} Whether there are any remaining operations. + */ + hasNext() { + return this._nextMatch && !!this._nextMatch[0]; + } + + /** + * Returns the next operation object and advances the iterator. + * + * Note: This does NOT implement the ECMAScript iterator protocol. + * + * @param {Op} [opOut] - Deprecated. Operation object to recycle for the return value. + * @returns {Op} The next operation, or an operation with a falsy `opcode` property if there are + * no more operations. + */ + next(opOut = new Op()) { + if (this.hasNext()) { + opOut.attribs = this._nextMatch[1]; + opOut.lines = exports.parseNum(this._nextMatch[2] || '0'); + opOut.opcode = this._nextMatch[3]; + opOut.chars = exports.parseNum(this._nextMatch[4]); + this._nextMatch = this._nextRegexMatch(); + } else { + clearOp(opOut); + } + return opOut; + } +} /** * Creates an iterator which decodes string changeset operations. @@ -196,38 +239,7 @@ exports.newLen = (cs) => exports.unpack(cs).newLen; * @param {string} opsStr - String encoding of the change operations to perform. * @returns {OpIter} Operator iterator object. */ -exports.opIterator = (opsStr) => { - const regex = /((?:\*[0-9a-z]+)*)(?:\|([0-9a-z]+))?([-+=])([0-9a-z]+)|(.)/g; - - const nextRegexMatch = () => { - const result = regex.exec(opsStr); - if (!result) return null; - if (result[5] === '$') return null; // Start of the insert operation character bank. - if (result[5] != null) error(`invalid operation: ${opsStr.slice(regex.lastIndex - 1)}`); - return result; - }; - let regexResult = nextRegexMatch(); - - const hasNext = () => regexResult && !!regexResult[0]; - - const next = (op = new Op()) => { - if (hasNext()) { - op.attribs = regexResult[1]; - op.lines = exports.parseNum(regexResult[2] || '0'); - op.opcode = regexResult[3]; - op.chars = exports.parseNum(regexResult[4]); - regexResult = nextRegexMatch(); - } else { - clearOp(op); - } - return op; - }; - - return { - next, - hasNext, - }; -}; +exports.opIterator = (opsStr) => new OpIter(opsStr); /** * Cleans an Op object. @@ -352,7 +364,7 @@ exports.checkRep = (cs) => { let oldPos = 0; let calcNewLen = 0; let numInserted = 0; - const iter = exports.opIterator(ops); + const iter = new OpIter(ops); while (iter.hasNext()) { const o = iter.next(); switch (o.opcode) { @@ -1005,8 +1017,8 @@ class TextLinesMutator { * @returns {string} the integrated changeset */ const applyZip = (in1, in2, func) => { - const iter1 = exports.opIterator(in1); - const iter2 = exports.opIterator(in2); + const iter1 = new OpIter(in1); + const iter2 = new OpIter(in2); const assem = exports.smartOpAssembler(); const op1 = new Op(); const op2 = new Op(); @@ -1075,7 +1087,7 @@ exports.pack = (oldLen, newLen, opsStr, bank) => { exports.applyToText = (cs, str) => { const unpacked = exports.unpack(cs); assert(str.length === unpacked.oldLen, `mismatched apply: ${str.length} / ${unpacked.oldLen}`); - const csIter = exports.opIterator(unpacked.ops); + const csIter = new OpIter(unpacked.ops); const bankIter = exports.stringIterator(unpacked.charBank); const strIter = exports.stringIterator(str); const assem = exports.stringAssembler(); @@ -1120,7 +1132,7 @@ exports.applyToText = (cs, str) => { */ exports.mutateTextLines = (cs, lines) => { const unpacked = exports.unpack(cs); - const csIter = exports.opIterator(unpacked.ops); + const csIter = new OpIter(unpacked.ops); const bankIter = exports.stringIterator(unpacked.charBank); const mut = new TextLinesMutator(lines); while (csIter.hasNext()) { @@ -1251,7 +1263,7 @@ exports.applyToAttribution = (cs, astr, pool) => { exports.mutateAttributionLines = (cs, lines, pool) => { const unpacked = exports.unpack(cs); - const csIter = exports.opIterator(unpacked.ops); + const csIter = new OpIter(unpacked.ops); const csBank = unpacked.charBank; let csBankIndex = 0; // treat the attribution lines as text lines, mutating a line at a time @@ -1265,7 +1277,7 @@ exports.mutateAttributionLines = (cs, lines, pool) => { const nextMutOp = () => { if ((!(lineIter && lineIter.hasNext())) && mut.hasMore()) { const line = mut.removeLines(1); - lineIter = exports.opIterator(line); + lineIter = new OpIter(line); } if (!lineIter || !lineIter.hasNext()) return new Op(); return lineIter.next(); @@ -1328,7 +1340,7 @@ exports.mutateAttributionLines = (cs, lines, pool) => { exports.joinAttributionLines = (theAlines) => { const assem = exports.mergingOpAssembler(); for (const aline of theAlines) { - const iter = exports.opIterator(aline); + const iter = new OpIter(aline); while (iter.hasNext()) { assem.append(iter.next()); } @@ -1337,7 +1349,7 @@ exports.joinAttributionLines = (theAlines) => { }; exports.splitAttributionLines = (attrOps, text) => { - const iter = exports.opIterator(attrOps); + const iter = new OpIter(attrOps); const assem = exports.mergingOpAssembler(); const lines = []; let pos = 0; @@ -1495,7 +1507,7 @@ const toSplices = (cs) => { const splices = []; let oldPos = 0; - const iter = exports.opIterator(unpacked.ops); + const iter = new OpIter(unpacked.ops); const charIter = exports.stringIterator(unpacked.charBank); let inSplice = false; while (iter.hasNext()) { @@ -1742,7 +1754,7 @@ exports.copyAText = (atext1, atext2) => { */ exports.opsFromAText = function* (atext) { // intentionally skips last newline char of atext - const iter = exports.opIterator(atext.attribs); + const iter = new OpIter(atext.attribs); let lastOp = null; while (iter.hasNext()) { if (lastOp != null) yield lastOp; @@ -1964,7 +1976,7 @@ exports.makeAttribsString = (opcode, attribs, pool) => { * Like "substring" but on a single-line attribution string. */ exports.subattribution = (astr, start, optEnd) => { - const iter = exports.opIterator(astr); + const iter = new OpIter(astr); const assem = exports.smartOpAssembler(); let attOp = new Op(); const csOp = new Op(); @@ -2033,13 +2045,13 @@ exports.inverse = (cs, lines, alines, pool) => { let curLineNextOp = new Op('+'); const unpacked = exports.unpack(cs); - const csIter = exports.opIterator(unpacked.ops); + const csIter = new OpIter(unpacked.ops); const builder = exports.builder(unpacked.newLen); const consumeAttribRuns = (numChars, func /* (len, attribs, endsLine)*/) => { if ((!curLineOpIter) || (curLineOpIterLine !== curLine)) { // create curLineOpIter and advance it to curChar - curLineOpIter = exports.opIterator(alinesGet(curLine)); + curLineOpIter = new OpIter(alinesGet(curLine)); curLineOpIterLine = curLine; let indexIntoLine = 0; while (curLineOpIter.hasNext()) { @@ -2058,7 +2070,7 @@ exports.inverse = (cs, lines, alines, pool) => { curChar = 0; curLineOpIterLine = curLine; curLineNextOp.chars = 0; - curLineOpIter = exports.opIterator(alinesGet(curLine)); + curLineOpIter = new OpIter(alinesGet(curLine)); } if (!curLineNextOp.chars) { curLineNextOp = curLineOpIter.hasNext() ? curLineOpIter.next() : new Op(); From 0eca0251f2abad825ae246b109686eb41ccc1676 Mon Sep 17 00:00:00 2001 From: Richard Hansen Date: Wed, 20 Oct 2021 20:34:09 -0400 Subject: [PATCH 027/446] Changeset: Use a generator to implement `OpIter` --- src/static/js/Changeset.js | 44 +++++++++++++++++++++++--------------- 1 file changed, 27 insertions(+), 17 deletions(-) diff --git a/src/static/js/Changeset.js b/src/static/js/Changeset.js index 2b0e82482..ac19f468a 100644 --- a/src/static/js/Changeset.js +++ b/src/static/js/Changeset.js @@ -180,6 +180,28 @@ exports.oldLen = (cs) => exports.unpack(cs).oldLen; */ exports.newLen = (cs) => exports.unpack(cs).newLen; +/** + * Parses a string of serialized changeset operations. + * + * @param {string} ops - Serialized changeset operations. + * @yields {Op} + * @returns {Generator} + */ +const deserializeOps = function* (ops) { + // TODO: Migrate to String.prototype.matchAll() once there is enough browser support. + const regex = /((?:\*[0-9a-z]+)*)(?:\|([0-9a-z]+))?([-+=])([0-9a-z]+)|(.)/g; + let match; + while ((match = regex.exec(ops)) != null) { + if (match[5] === '$') return; // Start of the insert operation character bank. + if (match[5] != null) error(`invalid operation: ${ops.slice(regex.lastIndex - 1)}`); + const op = new Op(match[3]); + op.lines = exports.parseNum(match[2] || '0'); + op.chars = exports.parseNum(match[4]); + op.attribs = match[1]; + yield op; + } +}; + /** * Iterator over a changeset's operations. * @@ -190,24 +212,15 @@ class OpIter { * @param {string} ops - String encoding the change operations to iterate over. */ constructor(ops) { - this._ops = ops; - this._regex = /((?:\*[0-9a-z]+)*)(?:\|([0-9a-z]+))?([-+=])([0-9a-z]+)|(.)/g; - this._nextMatch = this._nextRegexMatch(); - } - - _nextRegexMatch() { - const match = this._regex.exec(this._ops); - if (!match) return null; - if (match[5] === '$') return null; // Start of the insert operation character bank. - if (match[5] != null) error(`invalid operation: ${this._ops.slice(this._regex.lastIndex - 1)}`); - return match; + this._gen = deserializeOps(ops); + this._next = this._gen.next(); } /** * @returns {boolean} Whether there are any remaining operations. */ hasNext() { - return this._nextMatch && !!this._nextMatch[0]; + return !this._next.done; } /** @@ -221,11 +234,8 @@ class OpIter { */ next(opOut = new Op()) { if (this.hasNext()) { - opOut.attribs = this._nextMatch[1]; - opOut.lines = exports.parseNum(this._nextMatch[2] || '0'); - opOut.opcode = this._nextMatch[3]; - opOut.chars = exports.parseNum(this._nextMatch[4]); - this._nextMatch = this._nextRegexMatch(); + copyOp(this._next.value, opOut); + this._next = this._gen.next(); } else { clearOp(opOut); } From 89fe40e08083de5ea7a3d47047978eab1b688ed2 Mon Sep 17 00:00:00 2001 From: Richard Hansen Date: Mon, 25 Oct 2021 05:48:58 -0400 Subject: [PATCH 028/446] Changeset: Migrate from `OpIter` to `deserializeOps()` --- CHANGELOG.md | 2 + doc/api/hooks_server-side.md | 5 +- src/node/db/API.js | 4 +- src/node/handler/PadMessageHandler.js | 11 +- src/node/utils/ExportHelper.js | 5 +- src/node/utils/ExportHtml.js | 5 +- src/node/utils/ExportTxt.js | 5 +- src/node/utils/ImportHtml.js | 4 +- src/node/utils/padDiff.js | 58 +++++----- src/static/js/AttributeManager.js | 24 ++-- src/static/js/Changeset.js | 152 ++++++++++++++------------ src/static/js/ace2_inner.js | 23 +--- src/static/js/broadcast.js | 9 +- src/static/js/changesettracker.js | 5 +- src/static/js/linestylefilter.js | 6 +- src/tests/frontend/specs/easysync.js | 8 +- 16 files changed, 147 insertions(+), 179 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 03aab8890..a16f29c85 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -21,6 +21,8 @@ * `eachAttribNumber()` * `makeAttribsString()` * `opAttributeValue()` + * `opIterator()`: Deprecated in favor of the new `deserializeOps()` generator + function. * `appendATextToAssembler()`: Deprecated in favor of the new `opsFromAText()` generator function. * `newOp()`: Deprecated in favor of the new `Op` class. diff --git a/doc/api/hooks_server-side.md b/doc/api/hooks_server-side.md index 38cca9baa..aa19adec7 100644 --- a/doc/api/hooks_server-side.md +++ b/doc/api/hooks_server-side.md @@ -670,9 +670,8 @@ const Changeset = require('ep_etherpad-lite/static/js/Changeset'); exports.getLineHTMLForExport = async (hookName, context) => { if (!context.attribLine) return; - const opIter = Changeset.opIterator(context.attribLine); - if (!opIter.hasNext()) return; - const op = opIter.next(); + const [op] = Changeset.deserializeOps(context.attribLine); + if (op == null) return; const heading = AttributeMap.fromString(op.attribs, context.apool).get('heading'); if (!heading) return; context.lineContent = `<${heading}>${context.lineContent}`; diff --git a/src/node/db/API.js b/src/node/db/API.js index d48867558..0c3ddae46 100644 --- a/src/node/db/API.js +++ b/src/node/db/API.js @@ -527,12 +527,10 @@ exports.restoreRevision = async (padID, rev) => { atext.text += '\n'; const eachAttribRun = (attribs, func) => { - const attribsIter = Changeset.opIterator(attribs); let textIndex = 0; const newTextStart = 0; const newTextEnd = atext.text.length; - while (attribsIter.hasNext()) { - const op = attribsIter.next(); + for (const op of Changeset.deserializeOps(attribs)) { const nextIndex = textIndex + op.chars; if (!(nextIndex <= newTextStart || textIndex >= newTextEnd)) { func(Math.max(newTextStart, textIndex), Math.min(newTextEnd, nextIndex), op.attribs); diff --git a/src/node/handler/PadMessageHandler.js b/src/node/handler/PadMessageHandler.js index 868ca5b4a..0225b00d1 100644 --- a/src/node/handler/PadMessageHandler.js +++ b/src/node/handler/PadMessageHandler.js @@ -586,12 +586,7 @@ const handleUserChanges = async (socket, message) => { Changeset.checkRep(changeset); // Validate all added 'author' attribs to be the same value as the current user - const iterator = Changeset.opIterator(Changeset.unpack(changeset).ops); - let op; - - while (iterator.hasNext()) { - op = iterator.next(); - + for (const op of Changeset.deserializeOps(Changeset.unpack(changeset).ops)) { // + can add text with attribs // = can change or add attribs // - can have attribs, but they are discarded and don't show up in the attribs - @@ -741,10 +736,8 @@ const _correctMarkersInPad = (atext, apool) => { // collect char positions of line markers (e.g. bullets) in new atext // that aren't at the start of a line const badMarkers = []; - const iter = Changeset.opIterator(atext.attribs); let offset = 0; - while (iter.hasNext()) { - const op = iter.next(); + for (const op of Changeset.deserializeOps(atext.attribs)) { const attribs = AttributeMap.fromString(op.attribs, apool); const hasMarker = AttributeManager.lineAttributes.some((a) => attribs.has(a)); if (hasMarker) { diff --git a/src/node/utils/ExportHelper.js b/src/node/utils/ExportHelper.js index 401fad70b..7962476e8 100644 --- a/src/node/utils/ExportHelper.js +++ b/src/node/utils/ExportHelper.js @@ -52,9 +52,8 @@ exports._analyzeLine = (text, aline, apool) => { let lineMarker = 0; line.listLevel = 0; if (aline) { - const opIter = Changeset.opIterator(aline); - if (opIter.hasNext()) { - const op = opIter.next(); + const [op] = Changeset.deserializeOps(aline); + if (op != null) { const attribs = AttributeMap.fromString(op.attribs, apool); let listType = attribs.get('list'); if (listType) { diff --git a/src/node/utils/ExportHtml.js b/src/node/utils/ExportHtml.js index 800798f9c..3d5f4cc72 100644 --- a/src/node/utils/ExportHtml.js +++ b/src/node/utils/ExportHtml.js @@ -197,13 +197,12 @@ const getHTMLFromAtext = async (pad, atext, authorColors) => { return; } - const iter = Changeset.opIterator(Changeset.subattribution(attribs, idx, idx + numChars)); + const ops = Changeset.deserializeOps(Changeset.subattribution(attribs, idx, idx + numChars)); idx += numChars; // this iterates over every op string and decides which tags to open or to close // based on the attribs used - while (iter.hasNext()) { - const o = iter.next(); + for (const o of ops) { const usedAttribs = []; // mark all attribs as used diff --git a/src/node/utils/ExportTxt.js b/src/node/utils/ExportTxt.js index 1d7ce5469..9511dd0e7 100644 --- a/src/node/utils/ExportTxt.js +++ b/src/node/utils/ExportTxt.js @@ -76,11 +76,10 @@ const getTXTFromAtext = (pad, atext, authorColors) => { return; } - const iter = Changeset.opIterator(Changeset.subattribution(attribs, idx, idx + numChars)); + const ops = Changeset.deserializeOps(Changeset.subattribution(attribs, idx, idx + numChars)); idx += numChars; - while (iter.hasNext()) { - const o = iter.next(); + for (const o of ops) { let propChanged = false; for (const a of attributes.decodeAttribString(o.attribs)) { diff --git a/src/node/utils/ImportHtml.js b/src/node/utils/ImportHtml.js index 58b79f3a1..059d57af6 100644 --- a/src/node/utils/ImportHtml.js +++ b/src/node/utils/ImportHtml.js @@ -67,12 +67,10 @@ exports.setPadHTML = async (pad, html) => { const builder = Changeset.builder(1); // assemble each line into the builder - const attribsIter = Changeset.opIterator(newAttribs); let textIndex = 0; const newTextStart = 0; const newTextEnd = newText.length; - while (attribsIter.hasNext()) { - const op = attribsIter.next(); + for (const op of Changeset.deserializeOps(newAttribs)) { const nextIndex = textIndex + op.chars; if (!(nextIndex <= newTextStart || textIndex >= newTextEnd)) { const start = Math.max(newTextStart, textIndex); diff --git a/src/node/utils/padDiff.js b/src/node/utils/padDiff.js index d0c61633f..4ab276b4b 100644 --- a/src/node/utils/padDiff.js +++ b/src/node/utils/padDiff.js @@ -35,16 +35,10 @@ PadDiff.prototype._isClearAuthorship = function (changeset) { return false; } - // lets iterator over the operators - const iterator = Changeset.opIterator(unpacked.ops); - - // get the first operator, this should be a clear operator - const clearOperator = iterator.next(); + const [clearOperator, anotherOp] = Changeset.deserializeOps(unpacked.ops); // check if there is only one operator - if (iterator.hasNext() === true) { - return false; - } + if (anotherOp != null) return false; // check if this operator doesn't change text if (clearOperator.opcode !== '=') { @@ -212,7 +206,6 @@ PadDiff.prototype._extendChangesetWithAuthor = (changeset, author, apool) => { // unpack const unpacked = Changeset.unpack(changeset); - const iterator = Changeset.opIterator(unpacked.ops); const assem = Changeset.opAssembler(); // create deleted attribs @@ -220,10 +213,7 @@ PadDiff.prototype._extendChangesetWithAuthor = (changeset, author, apool) => { const deletedAttrib = apool.putAttrib(['removed', true]); const attribs = `*${Changeset.numToString(authorAttrib)}*${Changeset.numToString(deletedAttrib)}`; - // iteratore over the operators of the changeset - while (iterator.hasNext()) { - const operator = iterator.next(); - + for (const operator of Changeset.deserializeOps(unpacked.ops)) { if (operator.opcode === '-') { // this is a delete operator, extend it with the author operator.attribs = attribs; @@ -268,22 +258,23 @@ PadDiff.prototype._createDeletionChangeset = function (cs, startAText, apool) { let curLine = 0; let curChar = 0; - let curLineOpIter = null; - let curLineOpIterLine; + let curLineOps = null; + let curLineOpsNext = null; + let curLineOpsLine; let curLineNextOp = new Changeset.Op('+'); const unpacked = Changeset.unpack(cs); - const csIter = Changeset.opIterator(unpacked.ops); const builder = Changeset.builder(unpacked.newLen); const consumeAttribRuns = (numChars, func /* (len, attribs, endsLine)*/) => { - if ((!curLineOpIter) || (curLineOpIterLine !== curLine)) { - // create curLineOpIter and advance it to curChar - curLineOpIter = Changeset.opIterator(aLinesGet(curLine)); - curLineOpIterLine = curLine; + if (!curLineOps || curLineOpsLine !== curLine) { + curLineOps = Changeset.deserializeOps(aLinesGet(curLine)); + curLineOpsNext = curLineOps.next(); + curLineOpsLine = curLine; let indexIntoLine = 0; - while (curLineOpIter.hasNext()) { - curLineNextOp = curLineOpIter.next(); + while (!curLineOpsNext.done) { + curLineNextOp = curLineOpsNext.value; + curLineOpsNext = curLineOps.next(); if (indexIntoLine + curLineNextOp.chars >= curChar) { curLineNextOp.chars -= (curChar - indexIntoLine); break; @@ -293,16 +284,22 @@ PadDiff.prototype._createDeletionChangeset = function (cs, startAText, apool) { } while (numChars > 0) { - if ((!curLineNextOp.chars) && (!curLineOpIter.hasNext())) { + if (!curLineNextOp.chars && curLineOpsNext.done) { curLine++; curChar = 0; - curLineOpIterLine = curLine; + curLineOpsLine = curLine; curLineNextOp.chars = 0; - curLineOpIter = Changeset.opIterator(aLinesGet(curLine)); + curLineOps = Changeset.deserializeOps(aLinesGet(curLine)); + curLineOpsNext = curLineOps.next(); } if (!curLineNextOp.chars) { - curLineNextOp = curLineOpIter.hasNext() ? curLineOpIter.next() : new Changeset.Op(); + if (curLineOpsNext.done) { + curLineNextOp = new Changeset.Op(); + } else { + curLineNextOp = curLineOpsNext.value; + curLineOpsNext = curLineOps.next(); + } } const charsToUse = Math.min(numChars, curLineNextOp.chars); @@ -314,7 +311,7 @@ PadDiff.prototype._createDeletionChangeset = function (cs, startAText, apool) { curChar += charsToUse; } - if ((!curLineNextOp.chars) && (!curLineOpIter.hasNext())) { + if (!curLineNextOp.chars && curLineOpsNext.done) { curLine++; curChar = 0; } @@ -324,7 +321,7 @@ PadDiff.prototype._createDeletionChangeset = function (cs, startAText, apool) { if (L) { curLine += L; curChar = 0; - } else if (curLineOpIter && curLineOpIterLine === curLine) { + } else if (curLineOps && curLineOpsLine === curLine) { consumeAttribRuns(N, () => {}); } else { curChar += N; @@ -361,10 +358,7 @@ PadDiff.prototype._createDeletionChangeset = function (cs, startAText, apool) { }; }; - // iterate over all operators of this changeset - while (csIter.hasNext()) { - const csOp = csIter.next(); - + for (const csOp of Changeset.deserializeOps(unpacked.ops)) { if (csOp.opcode === '=') { const textBank = nextText(csOp.chars); diff --git a/src/static/js/AttributeManager.js b/src/static/js/AttributeManager.js index ebef74c07..f508af641 100644 --- a/src/static/js/AttributeManager.js +++ b/src/static/js/AttributeManager.js @@ -150,9 +150,9 @@ AttributeManager.prototype = _(AttributeManager.prototype).extend({ // get `attributeName` attribute of first char of line const aline = this.rep.alines[lineNum]; if (!aline) return ''; - const opIter = Changeset.opIterator(aline); - if (!opIter.hasNext()) return ''; - return AttributeMap.fromString(opIter.next().attribs, this.rep.apool).get(attributeName) || ''; + const [op] = Changeset.deserializeOps(aline); + if (op == null) return ''; + return AttributeMap.fromString(op.attribs, this.rep.apool).get(attributeName) || ''; }, /* @@ -163,9 +163,8 @@ AttributeManager.prototype = _(AttributeManager.prototype).extend({ // get attributes of first char of line const aline = this.rep.alines[lineNum]; if (!aline) return []; - const opIter = Changeset.opIterator(aline); - if (!opIter.hasNext()) return []; - const op = opIter.next(); + const [op] = Changeset.deserializeOps(aline); + if (op == null) return []; return [...attributes.attribsFromString(op.attribs, this.rep.apool)]; }, @@ -221,13 +220,8 @@ AttributeManager.prototype = _(AttributeManager.prototype).extend({ const end = selEnd[1]; let hasAttrib = true; - // Iterate over attribs on this line - - const opIter = Changeset.opIterator(rep.alines[lineNum]); let indexIntoLine = 0; - - while (opIter.hasNext()) { - const op = opIter.next(); + for (const op of Changeset.deserializeOps(rep.alines[lineNum])) { const opStartInLine = indexIntoLine; const opEndInLine = opStartInLine + op.chars; if (!hasIt(op.attribs)) { @@ -260,15 +254,11 @@ AttributeManager.prototype = _(AttributeManager.prototype).extend({ if (!aline) { return []; } - // iterate through all operations of a line - const opIter = Changeset.opIterator(aline); // we need to sum up how much characters each operations take until the wanted position let currentPointer = 0; - let currentOperation; - while (opIter.hasNext()) { - currentOperation = opIter.next(); + for (const currentOperation of Changeset.deserializeOps(aline)) { currentPointer += currentOperation.chars; if (currentPointer <= column) continue; return [...attributes.attribsFromString(currentOperation.attribs, this.rep.apool)]; diff --git a/src/static/js/Changeset.js b/src/static/js/Changeset.js index ac19f468a..669041ffb 100644 --- a/src/static/js/Changeset.js +++ b/src/static/js/Changeset.js @@ -187,7 +187,7 @@ exports.newLen = (cs) => exports.unpack(cs).newLen; * @yields {Op} * @returns {Generator} */ -const deserializeOps = function* (ops) { +exports.deserializeOps = function* (ops) { // TODO: Migrate to String.prototype.matchAll() once there is enough browser support. const regex = /((?:\*[0-9a-z]+)*)(?:\|([0-9a-z]+))?([-+=])([0-9a-z]+)|(.)/g; let match; @@ -206,13 +206,15 @@ const deserializeOps = function* (ops) { * Iterator over a changeset's operations. * * Note: This class does NOT implement the ECMAScript iterable or iterator protocols. + * + * @deprecated Use `deserializeOps` instead. */ class OpIter { /** * @param {string} ops - String encoding the change operations to iterate over. */ constructor(ops) { - this._gen = deserializeOps(ops); + this._gen = exports.deserializeOps(ops); this._next = this._gen.next(); } @@ -246,10 +248,15 @@ class OpIter { /** * Creates an iterator which decodes string changeset operations. * + * @deprecated Use `deserializeOps` instead. * @param {string} opsStr - String encoding of the change operations to perform. * @returns {OpIter} Operator iterator object. */ -exports.opIterator = (opsStr) => new OpIter(opsStr); +exports.opIterator = (opsStr) => { + padutils.warnWithStack( + 'Changeset.opIterator() is deprecated; use Changeset.deserializeOps() instead'); + return new OpIter(opsStr); +}; /** * Cleans an Op object. @@ -374,9 +381,7 @@ exports.checkRep = (cs) => { let oldPos = 0; let calcNewLen = 0; let numInserted = 0; - const iter = new OpIter(ops); - while (iter.hasNext()) { - const o = iter.next(); + for (const o of exports.deserializeOps(ops)) { switch (o.opcode) { case '=': oldPos += o.chars; @@ -1027,15 +1032,18 @@ class TextLinesMutator { * @returns {string} the integrated changeset */ const applyZip = (in1, in2, func) => { - const iter1 = new OpIter(in1); - const iter2 = new OpIter(in2); + const ops1 = exports.deserializeOps(in1); + const ops2 = exports.deserializeOps(in2); + let next1 = ops1.next(); + let next2 = ops2.next(); const assem = exports.smartOpAssembler(); - const op1 = new Op(); - const op2 = new Op(); - while (op1.opcode || iter1.hasNext() || op2.opcode || iter2.hasNext()) { - if ((!op1.opcode) && iter1.hasNext()) iter1.next(op1); - if ((!op2.opcode) && iter2.hasNext()) iter2.next(op2); - const opOut = func(op1, op2); + while (!next1.done || !next2.done) { + if (!next1.done && !next1.value.opcode) next1 = ops1.next(); + if (!next2.done && !next2.value.opcode) next2 = ops2.next(); + if (next1.value == null) next1.value = new Op(); + if (next2.value == null) next2.value = new Op(); + if (!next1.value.opcode && !next2.value.opcode) break; + const opOut = func(next1.value, next2.value); if (opOut && opOut.opcode) assem.append(opOut); } assem.endDocument(); @@ -1097,12 +1105,10 @@ exports.pack = (oldLen, newLen, opsStr, bank) => { exports.applyToText = (cs, str) => { const unpacked = exports.unpack(cs); assert(str.length === unpacked.oldLen, `mismatched apply: ${str.length} / ${unpacked.oldLen}`); - const csIter = new OpIter(unpacked.ops); const bankIter = exports.stringIterator(unpacked.charBank); const strIter = exports.stringIterator(str); const assem = exports.stringAssembler(); - while (csIter.hasNext()) { - const op = csIter.next(); + for (const op of exports.deserializeOps(unpacked.ops)) { switch (op.opcode) { case '+': // op is + and op.lines 0: no newlines must be in op.chars @@ -1142,11 +1148,9 @@ exports.applyToText = (cs, str) => { */ exports.mutateTextLines = (cs, lines) => { const unpacked = exports.unpack(cs); - const csIter = new OpIter(unpacked.ops); const bankIter = exports.stringIterator(unpacked.charBank); const mut = new TextLinesMutator(lines); - while (csIter.hasNext()) { - const op = csIter.next(); + for (const op of exports.deserializeOps(unpacked.ops)) { switch (op.opcode) { case '+': mut.insert(bankIter.take(op.chars), op.lines); @@ -1273,24 +1277,30 @@ exports.applyToAttribution = (cs, astr, pool) => { exports.mutateAttributionLines = (cs, lines, pool) => { const unpacked = exports.unpack(cs); - const csIter = new OpIter(unpacked.ops); + const csOps = exports.deserializeOps(unpacked.ops); + let csOpsNext = csOps.next(); const csBank = unpacked.charBank; let csBankIndex = 0; // treat the attribution lines as text lines, mutating a line at a time const mut = new TextLinesMutator(lines); - /** @type {?OpIter} */ - let lineIter = null; + /** @type {?Generator} */ + let lineOps = null; + let lineOpsNext = null; - const isNextMutOp = () => (lineIter && lineIter.hasNext()) || mut.hasMore(); + const lineOpsHasNext = () => lineOpsNext && !lineOpsNext.done; + const isNextMutOp = () => lineOpsHasNext() || mut.hasMore(); const nextMutOp = () => { - if ((!(lineIter && lineIter.hasNext())) && mut.hasMore()) { + if (!lineOpsHasNext() && mut.hasMore()) { const line = mut.removeLines(1); - lineIter = new OpIter(line); + lineOps = exports.deserializeOps(line); + lineOpsNext = lineOps.next(); } - if (!lineIter || !lineIter.hasNext()) return new Op(); - return lineIter.next(); + if (!lineOpsHasNext()) return new Op(); + const op = lineOpsNext.value; + lineOpsNext = lineOps.next(); + return op; }; let lineAssem = null; @@ -1308,12 +1318,15 @@ exports.mutateAttributionLines = (cs, lines, pool) => { let csOp = new Op(); let attOp = new Op(); - while (csOp.opcode || csIter.hasNext() || attOp.opcode || isNextMutOp()) { - if (!csOp.opcode && csIter.hasNext()) csOp = csIter.next(); - if ((!csOp.opcode) && (!attOp.opcode) && (!lineAssem) && (!(lineIter && lineIter.hasNext()))) { + while (csOp.opcode || !csOpsNext.done || attOp.opcode || isNextMutOp()) { + if (!csOp.opcode && !csOpsNext.done) { + csOp = csOpsNext.value; + csOpsNext = csOps.next(); + } + if (!csOp.opcode && !attOp.opcode && !lineAssem && !lineOpsHasNext()) { break; // done - } else if (csOp.opcode === '=' && csOp.lines > 0 && (!csOp.attribs) && - (!attOp.opcode) && (!lineAssem) && (!(lineIter && lineIter.hasNext()))) { + } else if (csOp.opcode === '=' && csOp.lines > 0 && !csOp.attribs && !attOp.opcode && + !lineAssem && !lineOpsHasNext()) { // skip multiple lines; this is what makes small changes not order of the document size mut.skipLines(csOp.lines); csOp.opcode = ''; @@ -1350,16 +1363,12 @@ exports.mutateAttributionLines = (cs, lines, pool) => { exports.joinAttributionLines = (theAlines) => { const assem = exports.mergingOpAssembler(); for (const aline of theAlines) { - const iter = new OpIter(aline); - while (iter.hasNext()) { - assem.append(iter.next()); - } + for (const op of exports.deserializeOps(aline)) assem.append(op); } return assem.toString(); }; exports.splitAttributionLines = (attrOps, text) => { - const iter = new OpIter(attrOps); const assem = exports.mergingOpAssembler(); const lines = []; let pos = 0; @@ -1373,8 +1382,7 @@ exports.splitAttributionLines = (attrOps, text) => { pos += op.chars; }; - while (iter.hasNext()) { - const op = iter.next(); + for (const op of exports.deserializeOps(attrOps)) { let numChars = op.chars; let numLines = op.lines; while (numLines > 1) { @@ -1517,11 +1525,9 @@ const toSplices = (cs) => { const splices = []; let oldPos = 0; - const iter = new OpIter(unpacked.ops); const charIter = exports.stringIterator(unpacked.charBank); let inSplice = false; - while (iter.hasNext()) { - const op = iter.next(); + for (const op of exports.deserializeOps(unpacked.ops)) { if (op.opcode === '=') { oldPos += op.chars; inSplice = false; @@ -1764,11 +1770,10 @@ exports.copyAText = (atext1, atext2) => { */ exports.opsFromAText = function* (atext) { // intentionally skips last newline char of atext - const iter = new OpIter(atext.attribs); let lastOp = null; - while (iter.hasNext()) { + for (const op of exports.deserializeOps(atext.attribs)) { if (lastOp != null) yield lastOp; - lastOp = iter.next(); + lastOp = op; } if (lastOp == null) return; // exclude final newline @@ -1986,15 +1991,19 @@ exports.makeAttribsString = (opcode, attribs, pool) => { * Like "substring" but on a single-line attribution string. */ exports.subattribution = (astr, start, optEnd) => { - const iter = new OpIter(astr); + const attOps = exports.deserializeOps(astr); + let attOpsNext = attOps.next(); const assem = exports.smartOpAssembler(); let attOp = new Op(); const csOp = new Op(); const doCsOp = () => { if (!csOp.chars) return; - while (csOp.opcode && (attOp.opcode || iter.hasNext())) { - if (!attOp.opcode) attOp = iter.next(); + while (csOp.opcode && (attOp.opcode || !attOpsNext.done)) { + if (!attOp.opcode) { + attOp = attOpsNext.value; + attOpsNext = attOps.next(); + } if (csOp.opcode && attOp.opcode && csOp.chars >= attOp.chars && attOp.lines > 0 && csOp.lines <= 0) { csOp.lines++; @@ -2013,7 +2022,10 @@ exports.subattribution = (astr, start, optEnd) => { if (attOp.opcode) { assem.append(attOp); } - while (iter.hasNext()) assem.append(iter.next()); + while (!attOpsNext.done) { + assem.append(attOpsNext.value); + attOpsNext = attOps.next(); + } } else { csOp.opcode = '='; csOp.chars = optEnd - start; @@ -2050,22 +2062,23 @@ exports.inverse = (cs, lines, alines, pool) => { let curLine = 0; let curChar = 0; - let curLineOpIter = null; - let curLineOpIterLine; + let curLineOps = null; + let curLineOpsNext = null; + let curLineOpsLine; let curLineNextOp = new Op('+'); const unpacked = exports.unpack(cs); - const csIter = new OpIter(unpacked.ops); const builder = exports.builder(unpacked.newLen); const consumeAttribRuns = (numChars, func /* (len, attribs, endsLine)*/) => { - if ((!curLineOpIter) || (curLineOpIterLine !== curLine)) { - // create curLineOpIter and advance it to curChar - curLineOpIter = new OpIter(alinesGet(curLine)); - curLineOpIterLine = curLine; + if (!curLineOps || curLineOpsLine !== curLine) { + curLineOps = exports.deserializeOps(alinesGet(curLine)); + curLineOpsNext = curLineOps.next(); + curLineOpsLine = curLine; let indexIntoLine = 0; - while (curLineOpIter.hasNext()) { - curLineNextOp = curLineOpIter.next(); + while (!curLineOpsNext.done) { + curLineNextOp = curLineOpsNext.value; + curLineOpsNext = curLineOps.next(); if (indexIntoLine + curLineNextOp.chars >= curChar) { curLineNextOp.chars -= (curChar - indexIntoLine); break; @@ -2075,15 +2088,21 @@ exports.inverse = (cs, lines, alines, pool) => { } while (numChars > 0) { - if ((!curLineNextOp.chars) && (!curLineOpIter.hasNext())) { + if (!curLineNextOp.chars && curLineOpsNext.done) { curLine++; curChar = 0; - curLineOpIterLine = curLine; + curLineOpsLine = curLine; curLineNextOp.chars = 0; - curLineOpIter = new OpIter(alinesGet(curLine)); + curLineOps = exports.deserializeOps(alinesGet(curLine)); + curLineOpsNext = curLineOps.next(); } if (!curLineNextOp.chars) { - curLineNextOp = curLineOpIter.hasNext() ? curLineOpIter.next() : new Op(); + if (curLineOpsNext.done) { + curLineNextOp = new Op(); + } else { + curLineNextOp = curLineOpsNext.value; + curLineOpsNext = curLineOps.next(); + } } const charsToUse = Math.min(numChars, curLineNextOp.chars); func(charsToUse, curLineNextOp.attribs, charsToUse === curLineNextOp.chars && @@ -2093,7 +2112,7 @@ exports.inverse = (cs, lines, alines, pool) => { curChar += charsToUse; } - if ((!curLineNextOp.chars) && (!curLineOpIter.hasNext())) { + if (!curLineNextOp.chars && curLineOpsNext.done) { curLine++; curChar = 0; } @@ -2103,7 +2122,7 @@ exports.inverse = (cs, lines, alines, pool) => { if (L) { curLine += L; curChar = 0; - } else if (curLineOpIter && curLineOpIterLine === curLine) { + } else if (curLineOps && curLineOpsLine === curLine) { consumeAttribRuns(N, () => {}); } else { curChar += N; @@ -2138,8 +2157,7 @@ exports.inverse = (cs, lines, alines, pool) => { }; }; - while (csIter.hasNext()) { - const csOp = csIter.next(); + for (const csOp of exports.deserializeOps(unpacked.ops)) { if (csOp.opcode === '=') { if (csOp.attribs) { const attribs = AttributeMap.fromString(csOp.attribs, pool); diff --git a/src/static/js/ace2_inner.js b/src/static/js/ace2_inner.js index 484205c06..77912057e 100644 --- a/src/static/js/ace2_inner.js +++ b/src/static/js/ace2_inner.js @@ -1576,13 +1576,8 @@ function Ace2Inner(editorInfo, cssManagers) { const end = selEnd[1]; let hasAttrib = true; - // Iterate over attribs on this line - - const opIter = Changeset.opIterator(rep.alines[lineNum]); let indexIntoLine = 0; - - while (opIter.hasNext()) { - const op = opIter.next(); + for (const op of Changeset.deserializeOps(rep.alines[lineNum])) { const opStartInLine = indexIntoLine; const opEndInLine = opStartInLine + op.chars; if (!hasIt(op.attribs)) { @@ -1615,7 +1610,6 @@ function Ace2Inner(editorInfo, cssManagers) { const selStartLine = rep.selStart[0]; const selEndLine = rep.selEnd[0]; for (let n = selStartLine; n <= selEndLine; n++) { - const opIter = Changeset.opIterator(rep.alines[n]); let indexIntoLine = 0; let selectionStartInLine = 0; if (documentAttributeManager.lineHasMarker(n)) { @@ -1628,8 +1622,7 @@ function Ace2Inner(editorInfo, cssManagers) { if (n === selEndLine) { selectionEndInLine = rep.selEnd[1]; } - while (opIter.hasNext()) { - const op = opIter.next(); + for (const op of Changeset.deserializeOps(rep.alines[n])) { const opStartInLine = indexIntoLine; const opEndInLine = opStartInLine + op.chars; if (!hasIt(op.attribs)) { @@ -1754,12 +1747,10 @@ function Ace2Inner(editorInfo, cssManagers) { }; const eachAttribRun = (attribs, func /* (startInNewText, endInNewText, attribs)*/) => { - const attribsIter = Changeset.opIterator(attribs); let textIndex = 0; const newTextStart = commonStart; const newTextEnd = newText.length - commonEnd - (shiftFinalNewlineToBeforeNewText ? 1 : 0); - while (attribsIter.hasNext()) { - const op = attribsIter.next(); + for (const op of Changeset.deserializeOps(attribs)) { const nextIndex = textIndex + op.chars; if (!(nextIndex <= newTextStart || textIndex >= newTextEnd)) { func(Math.max(newTextStart, textIndex), Math.min(newTextEnd, nextIndex), op.attribs); @@ -1873,9 +1864,7 @@ function Ace2Inner(editorInfo, cssManagers) { const attribRuns = (attribs) => { const lengs = []; const atts = []; - const iter = Changeset.opIterator(attribs); - while (iter.hasNext()) { - const op = iter.next(); + for (const op of Changeset.deserializeOps(attribs)) { lengs.push(op.chars); atts.push(op.attribs); } @@ -2619,9 +2608,7 @@ function Ace2Inner(editorInfo, cssManagers) { // TODO: There appears to be a race condition or so. const authorIds = new Set(); if (alineAttrs) { - const opIter = Changeset.opIterator(alineAttrs); - while (opIter.hasNext()) { - const op = opIter.next(); + for (const op of Changeset.deserializeOps(alineAttrs)) { const authorId = AttributeMap.fromString(op.attribs, apool).get('author'); if (authorId) authorIds.add(authorId); } diff --git a/src/static/js/broadcast.js b/src/static/js/broadcast.js index 5b19acf88..4014d5829 100644 --- a/src/static/js/broadcast.js +++ b/src/static/js/broadcast.js @@ -162,13 +162,8 @@ const loadBroadcastJS = (socket, sendSocketMsg, fireWhenAllScriptsAreLoaded, Bro // some chars are replaced (no attributes change and no length change) // test if there are keep ops at the start of the cs if (lineChanged === undefined) { - lineChanged = 0; - const opIter = Changeset.opIterator(Changeset.unpack(changeset).ops); - - if (opIter.hasNext()) { - const op = opIter.next(); - if (op.opcode === '=') lineChanged += op.lines; - } + const [op] = Changeset.deserializeOps(Changeset.unpack(changeset).ops); + lineChanged = op != null && op.opcode === '=' ? op.lines : 0; } const goToLineNumber = (lineNumber) => { diff --git a/src/static/js/changesettracker.js b/src/static/js/changesettracker.js index c45a253d4..30c70aa74 100644 --- a/src/static/js/changesettracker.js +++ b/src/static/js/changesettracker.js @@ -143,12 +143,9 @@ const makeChangesetTracker = (scheduler, apool, aceCallbacksProvider) => { // Sanitize authorship: Replace all author attributes with this user's author ID in case the // text was copied from another author. const cs = Changeset.unpack(userChangeset); - const iterator = Changeset.opIterator(cs.ops); - let op; const assem = Changeset.mergingOpAssembler(); - while (iterator.hasNext()) { - op = iterator.next(); + for (const op of Changeset.deserializeOps(cs.ops)) { if (op.opcode === '+') { const attribs = AttributeMap.fromString(op.attribs, apool); const oldAuthorId = attribs.get('author'); diff --git a/src/static/js/linestylefilter.js b/src/static/js/linestylefilter.js index 19751999c..632e6b3cc 100644 --- a/src/static/js/linestylefilter.js +++ b/src/static/js/linestylefilter.js @@ -98,11 +98,13 @@ linestylefilter.getLineStyleFilter = (lineLength, aline, textAndClassFunc, apool return classes.substring(1); }; - const attributionIter = Changeset.opIterator(aline); + const attrOps = Changeset.deserializeOps(aline); + let attrOpsNext = attrOps.next(); let nextOp, nextOpClasses; const goNextOp = () => { - nextOp = attributionIter.hasNext() ? attributionIter.next() : new Changeset.Op(); + nextOp = attrOpsNext.done ? new Changeset.Op() : attrOpsNext.value; + if (!attrOpsNext.done) attrOpsNext = attrOps.next(); nextOpClasses = (nextOp.opcode && attribsToClasses(nextOp.attribs)); }; goNextOp(); diff --git a/src/tests/frontend/specs/easysync.js b/src/tests/frontend/specs/easysync.js index fc56c62a1..5d066dfd9 100644 --- a/src/tests/frontend/specs/easysync.js +++ b/src/tests/frontend/specs/easysync.js @@ -31,17 +31,15 @@ const randInt = (maxValue) => Math.floor(Math.random() * maxValue); describe('easysync', function () { it('throughIterator', async function () { const x = '-c*3*4+6|3=az*asdf0*1*2*3+1=1-1+1*0+1=1-1+1|c=c-1'; - const iter = Changeset.opIterator(x); const assem = Changeset.opAssembler(); - while (iter.hasNext()) assem.append(iter.next()); + for (const op of Changeset.deserializeOps(x)) assem.append(op); expect(assem.toString()).to.equal(x); }); it('throughSmartAssembler', async function () { const x = '-c*3*4+6|3=az*asdf0*1*2*3+1=1-1+1*0+1=1-1+1|c=c-1'; - const iter = Changeset.opIterator(x); const assem = Changeset.smartOpAssembler(); - while (iter.hasNext()) assem.append(iter.next()); + for (const op of Changeset.deserializeOps(x)) assem.append(op); assem.endDocument(); expect(assem.toString()).to.equal(x); }); @@ -730,7 +728,7 @@ describe('easysync', function () { p.putAttrib(['name', 'david']); p.putAttrib(['color', 'green']); - const stringOp = (str) => Changeset.opIterator(str).next(); + const stringOp = (str) => Changeset.deserializeOps(str).next().value; expect(Changeset.opAttributeValue(stringOp('*0*1+1'), 'name', p)).to.equal('david'); expect(Changeset.opAttributeValue(stringOp('*0+1'), 'name', p)).to.equal('david'); From d3427240c666c1b7aff0964e5e55eb31d2866933 Mon Sep 17 00:00:00 2001 From: Richard Hansen Date: Tue, 23 Nov 2021 21:06:16 -0500 Subject: [PATCH 029/446] tests: Serve all of `src/tests/frontend/`, not just specs --- src/node/utils/Minify.js | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/node/utils/Minify.js b/src/node/utils/Minify.js index 58dba2097..2e8a2d960 100644 --- a/src/node/utils/Minify.js +++ b/src/node/utils/Minify.js @@ -166,8 +166,8 @@ const minify = async (req, res) => { filename = path.join('../node_modules/', library, libraryPath); } } - const [, spec] = /^plugins\/ep_etherpad-lite\/(tests\/frontend\/specs\/.*)/.exec(filename) || []; - if (spec != null) filename = `../${spec}`; + const [, testf] = /^plugins\/ep_etherpad-lite\/(tests\/frontend\/.*)/.exec(filename) || []; + if (testf != null) filename = `../${testf}`; const contentType = mime.lookup(filename); From 6a7b54313fe9721b370be2ecebd4c60dd55433fb Mon Sep 17 00:00:00 2001 From: Richard Hansen Date: Sat, 13 Nov 2021 20:02:52 -0500 Subject: [PATCH 030/446] easysync tests: Move shared helper functions to the top This will make it easier to split `easysync.js` into multiple files. --- src/tests/frontend/specs/easysync.js | 426 +++++++++++++-------------- 1 file changed, 213 insertions(+), 213 deletions(-) diff --git a/src/tests/frontend/specs/easysync.js b/src/tests/frontend/specs/easysync.js index 5d066dfd9..4f157ddfa 100644 --- a/src/tests/frontend/specs/easysync.js +++ b/src/tests/frontend/specs/easysync.js @@ -28,6 +28,219 @@ const AttributePool = require('../../../static/js/AttributePool'); const randInt = (maxValue) => Math.floor(Math.random() * maxValue); +const poolOrArray = (attribs) => { + if (attribs.getAttrib) { + return attribs; // it's already an attrib pool + } else { + // assume it's an array of attrib strings to be split and added + const p = new AttributePool(); + attribs.forEach((kv) => { + p.putAttrib(kv.split(',')); + }); + return p; + } +}; + +const randomInlineString = (len) => { + const assem = Changeset.stringAssembler(); + for (let i = 0; i < len; i++) { + assem.append(String.fromCharCode(randInt(26) + 97)); + } + return assem.toString(); +}; + +const randomMultiline = (approxMaxLines, approxMaxCols) => { + const numParts = randInt(approxMaxLines * 2) + 1; + const txt = Changeset.stringAssembler(); + txt.append(randInt(2) ? '\n' : ''); + for (let i = 0; i < numParts; i++) { + if ((i % 2) === 0) { + if (randInt(10)) { + txt.append(randomInlineString(randInt(approxMaxCols) + 1)); + } else { + txt.append('\n'); + } + } else { + txt.append('\n'); + } + } + return txt.toString(); +}; + +const randomStringOperation = (numCharsLeft) => { + let result; + switch (randInt(9)) { + case 0: + { + // insert char + result = { + insert: randomInlineString(1), + }; + break; + } + case 1: + { + // delete char + result = { + remove: 1, + }; + break; + } + case 2: + { + // skip char + result = { + skip: 1, + }; + break; + } + case 3: + { + // insert small + result = { + insert: randomInlineString(randInt(4) + 1), + }; + break; + } + case 4: + { + // delete small + result = { + remove: randInt(4) + 1, + }; + break; + } + case 5: + { + // skip small + result = { + skip: randInt(4) + 1, + }; + break; + } + case 6: + { + // insert multiline; + result = { + insert: randomMultiline(5, 20), + }; + break; + } + case 7: + { + // delete multiline + result = { + remove: Math.round(numCharsLeft * Math.random() * Math.random()), + }; + break; + } + case 8: + { + // skip multiline + result = { + skip: Math.round(numCharsLeft * Math.random() * Math.random()), + }; + break; + } + case 9: + { + // delete to end + result = { + remove: numCharsLeft, + }; + break; + } + case 10: + { + // skip to end + result = { + skip: numCharsLeft, + }; + break; + } + } + const maxOrig = numCharsLeft - 1; + if ('remove' in result) { + result.remove = Math.min(result.remove, maxOrig); + } else if ('skip' in result) { + result.skip = Math.min(result.skip, maxOrig); + } + return result; +}; + +const randomTwoPropAttribs = (opcode) => { + // assumes attrib pool like ['apple,','apple,true','banana,','banana,true'] + if (opcode === '-' || randInt(3)) { + return ''; + } else if (randInt(3)) { // eslint-disable-line no-dupe-else-if + if (opcode === '+' || randInt(2)) { + return `*${Changeset.numToString(randInt(2) * 2 + 1)}`; + } else { + return `*${Changeset.numToString(randInt(2) * 2)}`; + } + } else if (opcode === '+' || randInt(4) === 0) { + return '*1*3'; + } else { + return ['*0*2', '*0*3', '*1*2'][randInt(3)]; + } +}; + +const randomTestChangeset = (origText, withAttribs) => { + const charBank = Changeset.stringAssembler(); + let textLeft = origText; // always keep final newline + const outTextAssem = Changeset.stringAssembler(); + const opAssem = Changeset.smartOpAssembler(); + const oldLen = origText.length; + + const nextOp = new Changeset.Op(); + + const appendMultilineOp = (opcode, txt) => { + nextOp.opcode = opcode; + if (withAttribs) { + nextOp.attribs = randomTwoPropAttribs(opcode); + } + txt.replace(/\n|[^\n]+/g, (t) => { + if (t === '\n') { + nextOp.chars = 1; + nextOp.lines = 1; + opAssem.append(nextOp); + } else { + nextOp.chars = t.length; + nextOp.lines = 0; + opAssem.append(nextOp); + } + return ''; + }); + }; + + const doOp = () => { + const o = randomStringOperation(textLeft.length); + if (o.insert) { + const txt = o.insert; + charBank.append(txt); + outTextAssem.append(txt); + appendMultilineOp('+', txt); + } else if (o.skip) { + const txt = textLeft.substring(0, o.skip); + textLeft = textLeft.substring(o.skip); + outTextAssem.append(txt); + appendMultilineOp('=', txt); + } else if (o.remove) { + const txt = textLeft.substring(0, o.remove); + textLeft = textLeft.substring(o.remove); + appendMultilineOp('-', txt); + } + }; + + while (textLeft.length > 1) doOp(); + for (let i = 0; i < 5; i++) doOp(); // do some more (only insertions will happen) + const outText = `${outTextAssem.toString()}\n`; + opAssem.endDocument(); + const cs = Changeset.pack(oldLen, outText.length, opAssem.toString(), charBank.toString()); + Changeset.checkRep(cs); + return [cs, outText]; +}; + describe('easysync', function () { it('throughIterator', async function () { const x = '-c*3*4+6|3=az*asdf0*1*2*3+1=1-1+1*0+1=1-1+1|c=c-1'; @@ -176,19 +389,6 @@ describe('easysync', function () { ['skip', 1, 1, true], ], ['banana\n', 'cabbage\n', 'duffle\n']); - const poolOrArray = (attribs) => { - if (attribs.getAttrib) { - return attribs; // it's already an attrib pool - } else { - // assume it's an array of attrib strings to be split and added - const p = new AttributePool(); - attribs.forEach((kv) => { - p.putAttrib(kv.split(',')); - }); - return p; - } - }; - const runApplyToAttributionTest = (testId, attribs, cs, inAttr, outCorrect) => { it(`applyToAttribution#${testId}`, async function () { const p = poolOrArray(attribs); @@ -354,206 +554,6 @@ describe('easysync', function () { '|1+1', ]); - const randomInlineString = (len) => { - const assem = Changeset.stringAssembler(); - for (let i = 0; i < len; i++) { - assem.append(String.fromCharCode(randInt(26) + 97)); - } - return assem.toString(); - }; - - const randomMultiline = (approxMaxLines, approxMaxCols) => { - const numParts = randInt(approxMaxLines * 2) + 1; - const txt = Changeset.stringAssembler(); - txt.append(randInt(2) ? '\n' : ''); - for (let i = 0; i < numParts; i++) { - if ((i % 2) === 0) { - if (randInt(10)) { - txt.append(randomInlineString(randInt(approxMaxCols) + 1)); - } else { - txt.append('\n'); - } - } else { - txt.append('\n'); - } - } - return txt.toString(); - }; - - const randomStringOperation = (numCharsLeft) => { - let result; - switch (randInt(9)) { - case 0: - { - // insert char - result = { - insert: randomInlineString(1), - }; - break; - } - case 1: - { - // delete char - result = { - remove: 1, - }; - break; - } - case 2: - { - // skip char - result = { - skip: 1, - }; - break; - } - case 3: - { - // insert small - result = { - insert: randomInlineString(randInt(4) + 1), - }; - break; - } - case 4: - { - // delete small - result = { - remove: randInt(4) + 1, - }; - break; - } - case 5: - { - // skip small - result = { - skip: randInt(4) + 1, - }; - break; - } - case 6: - { - // insert multiline; - result = { - insert: randomMultiline(5, 20), - }; - break; - } - case 7: - { - // delete multiline - result = { - remove: Math.round(numCharsLeft * Math.random() * Math.random()), - }; - break; - } - case 8: - { - // skip multiline - result = { - skip: Math.round(numCharsLeft * Math.random() * Math.random()), - }; - break; - } - case 9: - { - // delete to end - result = { - remove: numCharsLeft, - }; - break; - } - case 10: - { - // skip to end - result = { - skip: numCharsLeft, - }; - break; - } - } - const maxOrig = numCharsLeft - 1; - if ('remove' in result) { - result.remove = Math.min(result.remove, maxOrig); - } else if ('skip' in result) { - result.skip = Math.min(result.skip, maxOrig); - } - return result; - }; - - const randomTwoPropAttribs = (opcode) => { - // assumes attrib pool like ['apple,','apple,true','banana,','banana,true'] - if (opcode === '-' || randInt(3)) { - return ''; - } else if (randInt(3)) { // eslint-disable-line no-dupe-else-if - if (opcode === '+' || randInt(2)) { - return `*${Changeset.numToString(randInt(2) * 2 + 1)}`; - } else { - return `*${Changeset.numToString(randInt(2) * 2)}`; - } - } else if (opcode === '+' || randInt(4) === 0) { - return '*1*3'; - } else { - return ['*0*2', '*0*3', '*1*2'][randInt(3)]; - } - }; - - const randomTestChangeset = (origText, withAttribs) => { - const charBank = Changeset.stringAssembler(); - let textLeft = origText; // always keep final newline - const outTextAssem = Changeset.stringAssembler(); - const opAssem = Changeset.smartOpAssembler(); - const oldLen = origText.length; - - const nextOp = new Changeset.Op(); - - const appendMultilineOp = (opcode, txt) => { - nextOp.opcode = opcode; - if (withAttribs) { - nextOp.attribs = randomTwoPropAttribs(opcode); - } - txt.replace(/\n|[^\n]+/g, (t) => { - if (t === '\n') { - nextOp.chars = 1; - nextOp.lines = 1; - opAssem.append(nextOp); - } else { - nextOp.chars = t.length; - nextOp.lines = 0; - opAssem.append(nextOp); - } - return ''; - }); - }; - - const doOp = () => { - const o = randomStringOperation(textLeft.length); - if (o.insert) { - const txt = o.insert; - charBank.append(txt); - outTextAssem.append(txt); - appendMultilineOp('+', txt); - } else if (o.skip) { - const txt = textLeft.substring(0, o.skip); - textLeft = textLeft.substring(o.skip); - outTextAssem.append(txt); - appendMultilineOp('=', txt); - } else if (o.remove) { - const txt = textLeft.substring(0, o.remove); - textLeft = textLeft.substring(o.remove); - appendMultilineOp('-', txt); - } - }; - - while (textLeft.length > 1) doOp(); - for (let i = 0; i < 5; i++) doOp(); // do some more (only insertions will happen) - const outText = `${outTextAssem.toString()}\n`; - opAssem.endDocument(); - const cs = Changeset.pack(oldLen, outText.length, opAssem.toString(), charBank.toString()); - Changeset.checkRep(cs); - return [cs, outText]; - }; - const testCompose = (randomSeed) => { it(`testCompose#${randomSeed}`, async function () { const p = new AttributePool(); From ec3833ab66d672076758cf29cff7e79068ad932f Mon Sep 17 00:00:00 2001 From: webzwo0i Date: Sun, 17 Oct 2021 23:57:47 +0200 Subject: [PATCH 031/446] easysync tests: Convert IIFE into a `describe()` --- src/tests/frontend/specs/easysync.js | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/tests/frontend/specs/easysync.js b/src/tests/frontend/specs/easysync.js index 4f157ddfa..9d5b47074 100644 --- a/src/tests/frontend/specs/easysync.js +++ b/src/tests/frontend/specs/easysync.js @@ -596,7 +596,7 @@ describe('easysync', function () { expect(cs12).to.equal('Z:2>1+1*0|1=2$x'); }); - (() => { + describe('followAttributes & composeAttributes', function () { const p = new AttributePool(); p.putAttrib(['x', '']); p.putAttrib(['x', 'abc']); @@ -623,7 +623,7 @@ describe('easysync', function () { testFollow('*0*1', '', '', '*0*1', '*0*1'); testFollow('*0*4', '*2*3', '*3', '*0', '*0*3'); testFollow('*0*4', '*2', '', '*0*4', '*0*4'); - })(); + }); const testFollow = (randomSeed) => { it(`testFollow#${randomSeed}`, async function () { From 310444f5d3424ed72cd15b2b29dc364994fbbd3e Mon Sep 17 00:00:00 2001 From: webzwo0i Date: Sun, 17 Oct 2021 23:57:47 +0200 Subject: [PATCH 032/446] easysync tests: Rename tests --- src/tests/frontend/specs/easysync.js | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/src/tests/frontend/specs/easysync.js b/src/tests/frontend/specs/easysync.js index 9d5b47074..6cd3cc6b7 100644 --- a/src/tests/frontend/specs/easysync.js +++ b/src/tests/frontend/specs/easysync.js @@ -242,14 +242,14 @@ const randomTestChangeset = (origText, withAttribs) => { }; describe('easysync', function () { - it('throughIterator', async function () { + it('opAssembler', async function () { const x = '-c*3*4+6|3=az*asdf0*1*2*3+1=1-1+1*0+1=1-1+1|c=c-1'; const assem = Changeset.opAssembler(); for (const op of Changeset.deserializeOps(x)) assem.append(op); expect(assem.toString()).to.equal(x); }); - it('throughSmartAssembler', async function () { + it('smartOpAssembler', async function () { const x = '-c*3*4+6|3=az*asdf0*1*2*3+1=1-1+1*0+1=1-1+1|c=c-1'; const assem = Changeset.smartOpAssembler(); for (const op of Changeset.deserializeOps(x)) assem.append(op); @@ -607,7 +607,7 @@ describe('easysync', function () { let n = 0; const testFollow = (a, b, afb, bfa, merge) => { - it(`testFollow manual #${++n}`, async function () { + it(`manual #${++n}`, async function () { expect(Changeset.exportedForTestingOnly.followAttributes(a, b, p)).to.equal(afb); expect(Changeset.exportedForTestingOnly.followAttributes(b, a, p)).to.equal(bfa); expect(Changeset.composeAttributes(a, afb, true, p)).to.equal(merge); @@ -723,7 +723,7 @@ describe('easysync', function () { testCharacterRangeFollow(10, 'Z:2>1+1$a', [0, 0], false, [1, 1]); testCharacterRangeFollow(11, 'Z:2>1+1$a', [0, 0], true, [0, 0]); - it('testOpAttributeValue', async function () { + it('opAttributeValue', async function () { const p = new AttributePool(); p.putAttrib(['name', 'david']); p.putAttrib(['color', 'green']); @@ -805,7 +805,7 @@ describe('easysync', function () { ], '*0*1'); const testSubattribution = (testId, astr, start, end, correctOutput) => { - it(`testSubattribution#${testId}`, async function () { + it(`subattribution#${testId}`, async function () { const str = Changeset.subattribution(astr, start, end); expect(str).to.equal(correctOutput); }); From 617515bcbb77f11cbf022dbda37069e5c45a3970 Mon Sep 17 00:00:00 2001 From: webzwo0i Date: Sun, 17 Oct 2021 23:57:47 +0200 Subject: [PATCH 033/446] easysync tests: Group related tests --- src/tests/frontend/specs/easysync.js | 880 ++++++++++++++------------- 1 file changed, 460 insertions(+), 420 deletions(-) diff --git a/src/tests/frontend/specs/easysync.js b/src/tests/frontend/specs/easysync.js index 6cd3cc6b7..00ca58f45 100644 --- a/src/tests/frontend/specs/easysync.js +++ b/src/tests/frontend/specs/easysync.js @@ -241,7 +241,7 @@ const randomTestChangeset = (origText, withAttribs) => { return [cs, outText]; }; -describe('easysync', function () { +describe('easysync-assembler', function () { it('opAssembler', async function () { const x = '-c*3*4+6|3=az*asdf0*1*2*3+1=1-1+1*0+1=1-1+1|c=c-1'; const assem = Changeset.opAssembler(); @@ -257,6 +257,225 @@ describe('easysync', function () { expect(assem.toString()).to.equal(x); }); + describe('append atext to assembler', function () { + const testAppendATextToAssembler = (testId, atext, correctOps) => { + it(`testAppendATextToAssembler#${testId}`, async function () { + const assem = Changeset.smartOpAssembler(); + for (const op of Changeset.opsFromAText(atext)) assem.append(op); + expect(assem.toString()).to.equal(correctOps); + }); + }; + + testAppendATextToAssembler(1, { + text: '\n', + attribs: '|1+1', + }, ''); + testAppendATextToAssembler(2, { + text: '\n\n', + attribs: '|2+2', + }, '|1+1'); + testAppendATextToAssembler(3, { + text: '\n\n', + attribs: '*x|2+2', + }, '*x|1+1'); + testAppendATextToAssembler(4, { + text: '\n\n', + attribs: '*x|1+1|1+1', + }, '*x|1+1'); + testAppendATextToAssembler(5, { + text: 'foo\n', + attribs: '|1+4', + }, '+3'); + testAppendATextToAssembler(6, { + text: '\nfoo\n', + attribs: '|2+5', + }, '|1+1+3'); + testAppendATextToAssembler(7, { + text: '\nfoo\n', + attribs: '*x|2+5', + }, '*x|1+1*x+3'); + testAppendATextToAssembler(8, { + text: '\n\n\nfoo\n', + attribs: '|2+2*x|2+5', + }, '|2+2*x|1+1*x+3'); + }); +}); + +describe('easysync-compose', function () { + describe('compose', function () { + const testCompose = (randomSeed) => { + it(`testCompose#${randomSeed}`, async function () { + const p = new AttributePool(); + + const startText = `${randomMultiline(10, 20)}\n`; + + const x1 = randomTestChangeset(startText); + const change1 = x1[0]; + const text1 = x1[1]; + + const x2 = randomTestChangeset(text1); + const change2 = x2[0]; + const text2 = x2[1]; + + const x3 = randomTestChangeset(text2); + const change3 = x3[0]; + const text3 = x3[1]; + + const change12 = Changeset.checkRep(Changeset.compose(change1, change2, p)); + const change23 = Changeset.checkRep(Changeset.compose(change2, change3, p)); + const change123 = Changeset.checkRep(Changeset.compose(change12, change3, p)); + const change123a = Changeset.checkRep(Changeset.compose(change1, change23, p)); + expect(change123a).to.equal(change123); + + expect(Changeset.applyToText(change12, startText)).to.equal(text2); + expect(Changeset.applyToText(change23, text1)).to.equal(text3); + expect(Changeset.applyToText(change123, startText)).to.equal(text3); + }); + }; + + for (let i = 0; i < 30; i++) testCompose(i); + }); + + describe('compose attributes', function () { + it('simpleComposeAttributesTest', async function () { + const p = new AttributePool(); + p.putAttrib(['bold', '']); + p.putAttrib(['bold', 'true']); + const cs1 = Changeset.checkRep('Z:2>1*1+1*1=1$x'); + const cs2 = Changeset.checkRep('Z:3>0*0|1=3$'); + const cs12 = Changeset.checkRep(Changeset.compose(cs1, cs2, p)); + expect(cs12).to.equal('Z:2>1+1*0|1=2$x'); + }); + }); +}); + +describe('easysync-follow', function () { + describe('follow & compose', function () { + const testFollow = (randomSeed) => { + it(`testFollow#${randomSeed}`, async function () { + const p = new AttributePool(); + + const startText = `${randomMultiline(10, 20)}\n`; + + const cs1 = randomTestChangeset(startText)[0]; + const cs2 = randomTestChangeset(startText)[0]; + + const afb = Changeset.checkRep(Changeset.follow(cs1, cs2, false, p)); + const bfa = Changeset.checkRep(Changeset.follow(cs2, cs1, true, p)); + + const merge1 = Changeset.checkRep(Changeset.compose(cs1, afb)); + const merge2 = Changeset.checkRep(Changeset.compose(cs2, bfa)); + + expect(merge2).to.equal(merge1); + }); + }; + + for (let i = 0; i < 30; i++) testFollow(i); + }); + + describe('followAttributes & composeAttributes', function () { + const p = new AttributePool(); + p.putAttrib(['x', '']); + p.putAttrib(['x', 'abc']); + p.putAttrib(['x', 'def']); + p.putAttrib(['y', '']); + p.putAttrib(['y', 'abc']); + p.putAttrib(['y', 'def']); + let n = 0; + + const testFollow = (a, b, afb, bfa, merge) => { + it(`manual #${++n}`, async function () { + expect(Changeset.exportedForTestingOnly.followAttributes(a, b, p)).to.equal(afb); + expect(Changeset.exportedForTestingOnly.followAttributes(b, a, p)).to.equal(bfa); + expect(Changeset.composeAttributes(a, afb, true, p)).to.equal(merge); + expect(Changeset.composeAttributes(b, bfa, true, p)).to.equal(merge); + }); + }; + + testFollow('', '', '', '', ''); + testFollow('*0', '', '', '*0', '*0'); + testFollow('*0', '*0', '', '', '*0'); + testFollow('*0', '*1', '', '*0', '*0'); + testFollow('*1', '*2', '', '*1', '*1'); + testFollow('*0*1', '', '', '*0*1', '*0*1'); + testFollow('*0*4', '*2*3', '*3', '*0', '*0*3'); + testFollow('*0*4', '*2', '', '*0*4', '*0*4'); + }); + + describe('chracterRangeFollow', function () { + const testCharacterRangeFollow = (testId, cs, oldRange, insertionsAfter, correctNewRange) => { + it(`testCharacterRangeFollow#${testId}`, async function () { + cs = Changeset.checkRep(cs); + expect(Changeset.characterRangeFollow(cs, oldRange[0], oldRange[1], insertionsAfter)) + .to.eql(correctNewRange); + }); + }; + + testCharacterRangeFollow(1, 'Z:z>9*0=1=4-3+9=1|1-4-4+1*0+a$123456789abcdefghijk', + [7, 10], false, [14, 15]); + testCharacterRangeFollow(2, 'Z:bc<6|x=b4|2-6$', [400, 407], false, [400, 401]); + testCharacterRangeFollow(3, 'Z:4>0-3+3$abc', [0, 3], false, [3, 3]); + testCharacterRangeFollow(4, 'Z:4>0-3+3$abc', [0, 3], true, [0, 0]); + testCharacterRangeFollow(5, 'Z:5>1+1=1-3+3$abcd', [1, 4], false, [5, 5]); + testCharacterRangeFollow(6, 'Z:5>1+1=1-3+3$abcd', [1, 4], true, [2, 2]); + testCharacterRangeFollow(7, 'Z:5>1+1=1-3+3$abcd', [0, 6], false, [1, 7]); + testCharacterRangeFollow(8, 'Z:5>1+1=1-3+3$abcd', [0, 3], false, [1, 2]); + testCharacterRangeFollow(9, 'Z:5>1+1=1-3+3$abcd', [2, 5], false, [5, 6]); + testCharacterRangeFollow(10, 'Z:2>1+1$a', [0, 0], false, [1, 1]); + testCharacterRangeFollow(11, 'Z:2>1+1$a', [0, 0], true, [0, 0]); + }); +}); + +describe('easysync-inverseRandom', function () { + describe('inverse random', function () { + const testInverseRandom = (randomSeed) => { + it(`testInverseRandom#${randomSeed}`, async function () { + const p = poolOrArray(['apple,', 'apple,true', 'banana,', 'banana,true']); + + const startText = `${randomMultiline(10, 20)}\n`; + const alines = + Changeset.splitAttributionLines(Changeset.makeAttribution(startText), startText); + const lines = startText.slice(0, -1).split('\n').map((s) => `${s}\n`); + + const stylifier = randomTestChangeset(startText, true)[0]; + + Changeset.mutateAttributionLines(stylifier, alines, p); + Changeset.mutateTextLines(stylifier, lines); + + const changeset = randomTestChangeset(lines.join(''), true)[0]; + const inverseChangeset = Changeset.inverse(changeset, lines, alines, p); + + const origLines = lines.slice(); + const origALines = alines.slice(); + + Changeset.mutateTextLines(changeset, lines); + Changeset.mutateAttributionLines(changeset, alines, p); + Changeset.mutateTextLines(inverseChangeset, lines); + Changeset.mutateAttributionLines(inverseChangeset, alines, p); + expect(lines).to.eql(origLines); + expect(alines).to.eql(origALines); + }); + }; + + for (let i = 0; i < 30; i++) testInverseRandom(i); + }); + + describe('inverse', function () { + const testInverse = (testId, cs, lines, alines, pool, correctOutput) => { + it(`testInverse#${testId}`, async function () { + pool = poolOrArray(pool); + const str = Changeset.inverse(Changeset.checkRep(cs), lines, alines, pool); + expect(str).to.equal(correctOutput); + }); + }; + + // take "FFFFTTTTT" and apply "-FT--FFTT", the inverse of which is "--F--TT--" + testInverse(1, 'Z:9>0=1*0=1*1=1=2*0=2*1|1=2$', null, + ['+4*1+5'], ['bold,', 'bold,true'], 'Z:9>0=2*0=1=2*1=2$'); + }); +}); + +describe('easysync-mutations', function () { const applyMutations = (mu, arrayOfArrays) => { arrayOfArrays.forEach((a) => { const result = mu[a[0]](...a.slice(1)); @@ -389,22 +608,6 @@ describe('easysync', function () { ['skip', 1, 1, true], ], ['banana\n', 'cabbage\n', 'duffle\n']); - const runApplyToAttributionTest = (testId, attribs, cs, inAttr, outCorrect) => { - it(`applyToAttribution#${testId}`, async function () { - const p = poolOrArray(attribs); - const result = Changeset.applyToAttribution(Changeset.checkRep(cs), inAttr, p); - expect(result).to.equal(outCorrect); - }); - }; - - // turn cactus\n into actusabcd\n - runApplyToAttributionTest(1, - ['bold,', 'bold,true'], 'Z:7>3-1*0=1*1=1=3+4$abcd', '+1*1+1|1+5', '+1*1+1|1+8'); - - // turn "david\ngreenspan\n" into "david\ngreen\n" - runApplyToAttributionTest(2, - ['bold,', 'bold,true'], 'Z:g<4*1|1=6*1=5-4$', '|2+g', '*1|1+6*1+5|1+1'); - it('mutatorHasMore', async function () { const lines = ['1\n', '2\n', '3\n', '4\n']; let mu; @@ -445,365 +648,268 @@ describe('easysync', function () { expect(mu.hasMore()).to.be(false); }); - const runMutateAttributionTest = (testId, attribs, cs, alines, outCorrect) => { - it(`runMutateAttributionTest#${testId}`, async function () { - const p = poolOrArray(attribs); - const alines2 = Array.prototype.slice.call(alines); - Changeset.mutateAttributionLines(Changeset.checkRep(cs), alines2, p); - expect(alines2).to.eql(outCorrect); - - const removeQuestionMarks = (a) => a.replace(/\?/g, ''); - const inMerged = Changeset.joinAttributionLines(alines.map(removeQuestionMarks)); - const correctMerged = Changeset.joinAttributionLines(outCorrect.map(removeQuestionMarks)); - const mergedResult = Changeset.applyToAttribution(cs, inMerged, p); - expect(mergedResult).to.equal(correctMerged); - }); - }; - - // turn 123\n 456\n 789\n into 123\n 456\n 789\n - runMutateAttributionTest(1, - ['bold,true'], 'Z:c>0|1=4=1*0=1$', ['|1+4', '|1+4', '|1+4'], ['|1+4', '+1*0+1|1+2', '|1+4']); - - // make a document bold - runMutateAttributionTest(2, - ['bold,true'], 'Z:c>0*0|3=c$', ['|1+4', '|1+4', '|1+4'], ['*0|1+4', '*0|1+4', '*0|1+4']); - - // clear bold on document - runMutateAttributionTest(3, - ['bold,', 'bold,true'], 'Z:c>0*0|3=c$', - ['*1+1+1*1+1|1+1', '+1*1+1|1+2', '*1+1+1*1+1|1+1'], ['|1+4', '|1+4', '|1+4']); - - // add a character on line 3 of a document with 5 blank lines, and make sure - // the optimization that skips purely-kept lines is working; if any attribution string - // with a '?' is parsed it will cause an error. - runMutateAttributionTest(4, - ['foo,bar', 'line,1', 'line,2', 'line,3', 'line,4', 'line,5'], - 'Z:5>1|2=2+1$x', ['?*1|1+1', '?*2|1+1', '*3|1+1', '?*4|1+1', '?*5|1+1'], - ['?*1|1+1', '?*2|1+1', '+1*3|1+1', '?*4|1+1', '?*5|1+1']); - - const testPoolWithChars = (() => { - const p = new AttributePool(); - p.putAttrib(['char', 'newline']); - for (let i = 1; i < 36; i++) { - p.putAttrib(['char', Changeset.numToString(i)]); - } - p.putAttrib(['char', '']); - return p; - })(); - - // based on runMutationTest#1 - runMutateAttributionTest(5, testPoolWithChars, - 'Z:11>7-2*t+1*u+1|2=b|2+a=2*b+1*o+1*t+1*0|1+1*b+1*u+1=3|1-3-6$tucream\npie\nbot\nbu', - [ - '*a+1*p+2*l+1*e+1*0|1+1', - '*b+1*a+1*n+1*a+1*n+1*a+1*0|1+1', - '*c+1*a+1*b+2*a+1*g+1*e+1*0|1+1', - '*d+1*u+1*f+2*l+1*e+1*0|1+1', - '*e+1*g+2*p+1*l+1*a+1*n+1*t+1*0|1+1', - ], - [ - '*t+1*u+1*p+1*l+1*e+1*0|1+1', - '*b+1*a+1*n+1*a+1*n+1*a+1*0|1+1', - '|1+6', - '|1+4', - '*c+1*a+1*b+1*o+1*t+1*0|1+1', - '*b+1*u+1*b+2*a+1*0|1+1', - '*e+1*g+2*p+1*l+1*a+1*n+1*t+1*0|1+1', - ]); - - // based on runMutationTest#3 - runMutateAttributionTest(6, testPoolWithChars, - 'Z:117=1|4+7$\n2\n3\n4\n', - ['*1+1*5|1+2'], ['*1+1|1+1', '|1+2', '|1+2', '|1+2', '*5|1+2']); - - // based on runMutationTest#5 - runMutateAttributionTest(8, testPoolWithChars, 'Z:a<7=1|4-7$', - ['*1|1+2', '*2|1+2', '*3|1+2', '*4|1+2', '*5|1+2'], ['*1+1*5|1+2']); - - // based on runMutationTest#6 - runMutateAttributionTest(9, testPoolWithChars, 'Z:k<7*0+1*10|2=8|2-8$0', - [ - '*1+1*2+1*3+1|1+1', - '*a+1*b+1*c+1|1+1', - '*d+1*e+1*f+1|1+1', - '*g+1*h+1*i+1|1+1', - '?*x+1*y+1*z+1|1+1', - ], - ['*0+1|1+4', '|1+4', '?*x+1*y+1*z+1|1+1']); - - runMutateAttributionTest(10, testPoolWithChars, 'Z:6>4=1+1=1+1|1=1+1=1*0+1$abcd', - ['|1+3', '|1+3'], ['|1+5', '+2*0+1|1+2']); - - - runMutateAttributionTest(11, testPoolWithChars, 'Z:s>1|1=4=6|1+1$\n', - ['*0|1+4', '*0|1+8', '*0+5|1+1', '*0|1+1', '*0|1+5', '*0|1+1', '*0|1+1', '*0|1+1', '|1+1'], - [ - '*0|1+4', - '*0+6|1+1', - '*0|1+2', - '*0+5|1+1', - '*0|1+1', - '*0|1+5', - '*0|1+1', - '*0|1+1', - '*0|1+1', - '|1+1', - ]); - - const testCompose = (randomSeed) => { - it(`testCompose#${randomSeed}`, async function () { - const p = new AttributePool(); - - const startText = `${randomMultiline(10, 20)}\n`; - - const x1 = randomTestChangeset(startText); - const change1 = x1[0]; - const text1 = x1[1]; - - const x2 = randomTestChangeset(text1); - const change2 = x2[0]; - const text2 = x2[1]; - - const x3 = randomTestChangeset(text2); - const change3 = x3[0]; - const text3 = x3[1]; - - const change12 = Changeset.checkRep(Changeset.compose(change1, change2, p)); - const change23 = Changeset.checkRep(Changeset.compose(change2, change3, p)); - const change123 = Changeset.checkRep(Changeset.compose(change12, change3, p)); - const change123a = Changeset.checkRep(Changeset.compose(change1, change23, p)); - expect(change123a).to.equal(change123); - - expect(Changeset.applyToText(change12, startText)).to.equal(text2); - expect(Changeset.applyToText(change23, text1)).to.equal(text3); - expect(Changeset.applyToText(change123, startText)).to.equal(text3); - }); - }; - - for (let i = 0; i < 30; i++) testCompose(i); - - it('simpleComposeAttributesTest', async function () { - const p = new AttributePool(); - p.putAttrib(['bold', '']); - p.putAttrib(['bold', 'true']); - const cs1 = Changeset.checkRep('Z:2>1*1+1*1=1$x'); - const cs2 = Changeset.checkRep('Z:3>0*0|1=3$'); - const cs12 = Changeset.checkRep(Changeset.compose(cs1, cs2, p)); - expect(cs12).to.equal('Z:2>1+1*0|1=2$x'); - }); - - describe('followAttributes & composeAttributes', function () { - const p = new AttributePool(); - p.putAttrib(['x', '']); - p.putAttrib(['x', 'abc']); - p.putAttrib(['x', 'def']); - p.putAttrib(['y', '']); - p.putAttrib(['y', 'abc']); - p.putAttrib(['y', 'def']); - let n = 0; - - const testFollow = (a, b, afb, bfa, merge) => { - it(`manual #${++n}`, async function () { - expect(Changeset.exportedForTestingOnly.followAttributes(a, b, p)).to.equal(afb); - expect(Changeset.exportedForTestingOnly.followAttributes(b, a, p)).to.equal(bfa); - expect(Changeset.composeAttributes(a, afb, true, p)).to.equal(merge); - expect(Changeset.composeAttributes(b, bfa, true, p)).to.equal(merge); + describe('mutateTextLines', function () { + const testMutateTextLines = (testId, cs, lines, correctLines) => { + it(`testMutateTextLines#${testId}`, async function () { + const a = lines.slice(); + Changeset.mutateTextLines(cs, a); + expect(a).to.eql(correctLines); }); }; - testFollow('', '', '', '', ''); - testFollow('*0', '', '', '*0', '*0'); - testFollow('*0', '*0', '', '', '*0'); - testFollow('*0', '*1', '', '*0', '*0'); - testFollow('*1', '*2', '', '*1', '*1'); - testFollow('*0*1', '', '', '*0*1', '*0*1'); - testFollow('*0*4', '*2*3', '*3', '*0', '*0*3'); - testFollow('*0*4', '*2', '', '*0*4', '*0*4'); + testMutateTextLines(1, 'Z:4<1|1-2-1|1+1+1$\nc', ['a\n', 'b\n'], ['\n', 'c\n']); + testMutateTextLines(2, 'Z:4>0|1-2-1|2+3$\nc\n', ['a\n', 'b\n'], ['\n', 'c\n', '\n']); }); - const testFollow = (randomSeed) => { - it(`testFollow#${randomSeed}`, async function () { + describe('mutate attributions', function () { + const testPoolWithChars = (() => { const p = new AttributePool(); - - const startText = `${randomMultiline(10, 20)}\n`; - - const cs1 = randomTestChangeset(startText)[0]; - const cs2 = randomTestChangeset(startText)[0]; - - const afb = Changeset.checkRep(Changeset.follow(cs1, cs2, false, p)); - const bfa = Changeset.checkRep(Changeset.follow(cs2, cs1, true, p)); - - const merge1 = Changeset.checkRep(Changeset.compose(cs1, afb)); - const merge2 = Changeset.checkRep(Changeset.compose(cs2, bfa)); - - expect(merge2).to.equal(merge1); - }); - }; - - for (let i = 0; i < 30; i++) testFollow(i); - - const testSplitJoinAttributionLines = (randomSeed) => { - const stringToOps = (str) => { - const assem = Changeset.mergingOpAssembler(); - const o = new Changeset.Op('+'); - o.chars = 1; - for (let i = 0; i < str.length; i++) { - const c = str.charAt(i); - o.lines = (c === '\n' ? 1 : 0); - o.attribs = (c === 'a' || c === 'b' ? `*${c}` : ''); - assem.append(o); + p.putAttrib(['char', 'newline']); + for (let i = 1; i < 36; i++) { + p.putAttrib(['char', Changeset.numToString(i)]); } - return assem.toString(); + p.putAttrib(['char', '']); + return p; + })(); + + const runMutateAttributionTest = (testId, attribs, cs, alines, outCorrect) => { + it(`runMutateAttributionTest#${testId}`, async function () { + const p = poolOrArray(attribs); + const alines2 = Array.prototype.slice.call(alines); + Changeset.mutateAttributionLines(Changeset.checkRep(cs), alines2, p); + expect(alines2).to.eql(outCorrect); + + const removeQuestionMarks = (a) => a.replace(/\?/g, ''); + const inMerged = Changeset.joinAttributionLines(alines.map(removeQuestionMarks)); + const correctMerged = Changeset.joinAttributionLines(outCorrect.map(removeQuestionMarks)); + const mergedResult = Changeset.applyToAttribution(cs, inMerged, p); + expect(mergedResult).to.equal(correctMerged); + }); }; - it(`testSplitJoinAttributionLines#${randomSeed}`, async function () { - const doc = `${randomMultiline(10, 20)}\n`; + // turn 123\n 456\n 789\n into 123\n 456\n 789\n + runMutateAttributionTest(1, + ['bold,true'], 'Z:c>0|1=4=1*0=1$', ['|1+4', '|1+4', '|1+4'], ['|1+4', '+1*0+1|1+2', '|1+4'] + ); - const theJoined = stringToOps(doc); - const theSplit = doc.match(/[^\n]*\n/g).map(stringToOps); + // make a document bold + runMutateAttributionTest(2, + ['bold,true'], 'Z:c>0*0|3=c$', ['|1+4', '|1+4', '|1+4'], ['*0|1+4', '*0|1+4', '*0|1+4']); - expect(Changeset.splitAttributionLines(theJoined, doc)).to.eql(theSplit); - expect(Changeset.joinAttributionLines(theSplit)).to.equal(theJoined); - }); - }; + // clear bold on document + runMutateAttributionTest(3, + ['bold,', 'bold,true'], 'Z:c>0*0|3=c$', + ['*1+1+1*1+1|1+1', '+1*1+1|1+2', '*1+1+1*1+1|1+1'], ['|1+4', '|1+4', '|1+4']); - for (let i = 0; i < 10; i++) testSplitJoinAttributionLines(i); + // add a character on line 3 of a document with 5 blank lines, and make sure + // the optimization that skips purely-kept lines is working; if any attribution string + // with a '?' is parsed it will cause an error. + runMutateAttributionTest(4, + ['foo,bar', 'line,1', 'line,2', 'line,3', 'line,4', 'line,5'], + 'Z:5>1|2=2+1$x', ['?*1|1+1', '?*2|1+1', '*3|1+1', '?*4|1+1', '?*5|1+1'], + ['?*1|1+1', '?*2|1+1', '+1*3|1+1', '?*4|1+1', '?*5|1+1']); - it('testMoveOpsToNewPool', async function () { - const pool1 = new AttributePool(); - const pool2 = new AttributePool(); + // based on runMutationTest#1 + runMutateAttributionTest(5, testPoolWithChars, + 'Z:11>7-2*t+1*u+1|2=b|2+a=2*b+1*o+1*t+1*0|1+1*b+1*u+1=3|1-3-6$tucream\npie\nbot\nbu', + [ + '*a+1*p+2*l+1*e+1*0|1+1', + '*b+1*a+1*n+1*a+1*n+1*a+1*0|1+1', + '*c+1*a+1*b+2*a+1*g+1*e+1*0|1+1', + '*d+1*u+1*f+2*l+1*e+1*0|1+1', + '*e+1*g+2*p+1*l+1*a+1*n+1*t+1*0|1+1', + ], + [ + '*t+1*u+1*p+1*l+1*e+1*0|1+1', + '*b+1*a+1*n+1*a+1*n+1*a+1*0|1+1', + '|1+6', + '|1+4', + '*c+1*a+1*b+1*o+1*t+1*0|1+1', + '*b+1*u+1*b+2*a+1*0|1+1', + '*e+1*g+2*p+1*l+1*a+1*n+1*t+1*0|1+1', + ]); - pool1.putAttrib(['baz', 'qux']); - pool1.putAttrib(['foo', 'bar']); + // based on runMutationTest#3 + runMutateAttributionTest(6, testPoolWithChars, + 'Z:117=1|4+7$\n2\n3\n4\n', + ['*1+1*5|1+2'], ['*1+1|1+1', '|1+2', '|1+2', '|1+2', '*5|1+2']); - expect(Changeset.moveOpsToNewPool('Z:1>2*1+1*0+1$ab', pool1, pool2)) - .to.equal('Z:1>2*0+1*1+1$ab'); - expect(Changeset.moveOpsToNewPool('*1+1*0+1', pool1, pool2)).to.equal('*0+1*1+1'); + // based on runMutationTest#5 + runMutateAttributionTest(8, testPoolWithChars, 'Z:a<7=1|4-7$', + ['*1|1+2', '*2|1+2', '*3|1+2', '*4|1+2', '*5|1+2'], ['*1+1*5|1+2']); + + // based on runMutationTest#6 + runMutateAttributionTest(9, testPoolWithChars, 'Z:k<7*0+1*10|2=8|2-8$0', + [ + '*1+1*2+1*3+1|1+1', + '*a+1*b+1*c+1|1+1', + '*d+1*e+1*f+1|1+1', + '*g+1*h+1*i+1|1+1', + '?*x+1*y+1*z+1|1+1', + ], + ['*0+1|1+4', '|1+4', '?*x+1*y+1*z+1|1+1']); + + runMutateAttributionTest(10, testPoolWithChars, 'Z:6>4=1+1=1+1|1=1+1=1*0+1$abcd', + ['|1+3', '|1+3'], ['|1+5', '+2*0+1|1+2']); + + runMutateAttributionTest(11, testPoolWithChars, 'Z:s>1|1=4=6|1+1$\n', + ['*0|1+4', '*0|1+8', '*0+5|1+1', '*0|1+1', '*0|1+5', '*0|1+1', '*0|1+1', '*0|1+1', '|1+1'], + [ + '*0|1+4', + '*0+6|1+1', + '*0|1+2', + '*0+5|1+1', + '*0|1+1', + '*0|1+5', + '*0|1+1', + '*0|1+1', + '*0|1+1', + '|1+1', + ]); + }); +}); + +describe('easysync-other', function () { + describe('filter attribute numbers', function () { + const testFilterAttribNumbers = (testId, cs, filter, correctOutput) => { + it(`testFilterAttribNumbers#${testId}`, async function () { + const str = Changeset.filterAttribNumbers(cs, filter); + expect(str).to.equal(correctOutput); + }); + }; + + testFilterAttribNumbers(1, '*0*1+1+2+3*1+4*2+5*0*2*1*b*c+6', + (n) => (n % 2) === 0, '*0+1+2+3+4*2+5*0*2*c+6'); + testFilterAttribNumbers(2, '*0*1+1+2+3*1+4*2+5*0*2*1*b*c+6', + (n) => (n % 2) === 1, '*1+1+2+3*1+4+5*1*b+6'); }); - it('testMakeSplice', async function () { - const t = 'a\nb\nc\n'; - const t2 = Changeset.applyToText(Changeset.makeSplice(t, 5, 0, 'def'), t); - expect(t2).to.equal('a\nb\ncdef\n'); + describe('make attribs string', function () { + const testMakeAttribsString = (testId, pool, opcode, attribs, correctString) => { + it(`testMakeAttribsString#${testId}`, async function () { + const p = poolOrArray(pool); + const str = Changeset.makeAttribsString(opcode, attribs, p); + expect(str).to.equal(correctString); + }); + }; + + testMakeAttribsString(1, ['bold,'], '+', [ + ['bold', ''], + ], ''); + testMakeAttribsString(2, ['abc,def', 'bold,'], '=', [ + ['bold', ''], + ], '*1'); + testMakeAttribsString(3, ['abc,def', 'bold,true'], '+', [ + ['abc', 'def'], + ['bold', 'true'], + ], '*0*1'); + testMakeAttribsString(4, ['abc,def', 'bold,true'], '+', [ + ['bold', 'true'], + ['abc', 'def'], + ], '*0*1'); }); - it('testToSplices', async function () { - const cs = Changeset.checkRep('Z:z>9*0=1=4-3+9=1|1-4-4+1*0+a$123456789abcdefghijk'); - const correctSplices = [ - [5, 8, '123456789'], - [9, 17, 'abcdefghijk'], - ]; - expect(Changeset.exportedForTestingOnly.toSplices(cs)).to.eql(correctSplices); + describe('other', function () { + it('testMoveOpsToNewPool', async function () { + const pool1 = new AttributePool(); + const pool2 = new AttributePool(); + + pool1.putAttrib(['baz', 'qux']); + pool1.putAttrib(['foo', 'bar']); + + pool2.putAttrib(['foo', 'bar']); + + expect(Changeset.moveOpsToNewPool('Z:1>2*1+1*0+1$ab', pool1, pool2)) + .to.equal('Z:1>2*0+1*1+1$ab'); + expect(Changeset.moveOpsToNewPool('*1+1*0+1', pool1, pool2)).to.equal('*0+1*1+1'); + }); + + it('testMakeSplice', async function () { + const t = 'a\nb\nc\n'; + const t2 = Changeset.applyToText(Changeset.makeSplice(t, 5, 0, 'def'), t); + expect(t2).to.equal('a\nb\ncdef\n'); + }); + + it('testToSplices', async function () { + const cs = Changeset.checkRep('Z:z>9*0=1=4-3+9=1|1-4-4+1*0+a$123456789abcdefghijk'); + const correctSplices = [ + [5, 8, '123456789'], + [9, 17, 'abcdefghijk'], + ]; + expect(Changeset.exportedForTestingOnly.toSplices(cs)).to.eql(correctSplices); + }); + + it('opAttributeValue', async function () { + const p = new AttributePool(); + p.putAttrib(['name', 'david']); + p.putAttrib(['color', 'green']); + + const stringOp = (str) => Changeset.deserializeOps(str).next().value; + + expect(Changeset.opAttributeValue(stringOp('*0*1+1'), 'name', p)).to.equal('david'); + expect(Changeset.opAttributeValue(stringOp('*0+1'), 'name', p)).to.equal('david'); + expect(Changeset.opAttributeValue(stringOp('*1+1'), 'name', p)).to.equal(''); + expect(Changeset.opAttributeValue(stringOp('+1'), 'name', p)).to.equal(''); + expect(Changeset.opAttributeValue(stringOp('*0*1+1'), 'color', p)).to.equal('green'); + expect(Changeset.opAttributeValue(stringOp('*1+1'), 'color', p)).to.equal('green'); + expect(Changeset.opAttributeValue(stringOp('*0+1'), 'color', p)).to.equal(''); + expect(Changeset.opAttributeValue(stringOp('+1'), 'color', p)).to.equal(''); + }); + + describe('applyToAttribution', function () { + const runApplyToAttributionTest = (testId, attribs, cs, inAttr, outCorrect) => { + it(`applyToAttribution#${testId}`, async function () { + const p = poolOrArray(attribs); + const result = Changeset.applyToAttribution(Changeset.checkRep(cs), inAttr, p); + expect(result).to.equal(outCorrect); + }); + }; + + // turn cactus\n into actusabcd\n + runApplyToAttributionTest(1, + ['bold,', 'bold,true'], 'Z:7>3-1*0=1*1=1=3+4$abcd', '+1*1+1|1+5', '+1*1+1|1+8'); + + // turn "david\ngreenspan\n" into "david\ngreen\n" + runApplyToAttributionTest(2, + ['bold,', 'bold,true'], 'Z:g<4*1|1=6*1=5-4$', '|2+g', '*1|1+6*1+5|1+1'); + }); + + describe('split/join attribution lines', function () { + const testSplitJoinAttributionLines = (randomSeed) => { + const stringToOps = (str) => { + const assem = Changeset.mergingOpAssembler(); + const o = new Changeset.Op('+'); + o.chars = 1; + for (let i = 0; i < str.length; i++) { + const c = str.charAt(i); + o.lines = (c === '\n' ? 1 : 0); + o.attribs = (c === 'a' || c === 'b' ? `*${c}` : ''); + assem.append(o); + } + return assem.toString(); + }; + + it(`testSplitJoinAttributionLines#${randomSeed}`, async function () { + const doc = `${randomMultiline(10, 20)}\n`; + + const theJoined = stringToOps(doc); + const theSplit = doc.match(/[^\n]*\n/g).map(stringToOps); + + expect(Changeset.splitAttributionLines(theJoined, doc)).to.eql(theSplit); + expect(Changeset.joinAttributionLines(theSplit)).to.equal(theJoined); + }); + }; + + for (let i = 0; i < 10; i++) testSplitJoinAttributionLines(i); + }); }); +}); - const testCharacterRangeFollow = (testId, cs, oldRange, insertionsAfter, correctNewRange) => { - it(`testCharacterRangeFollow#${testId}`, async function () { - cs = Changeset.checkRep(cs); - expect(Changeset.characterRangeFollow(cs, oldRange[0], oldRange[1], insertionsAfter)) - .to.eql(correctNewRange); - }); - }; - - testCharacterRangeFollow(1, 'Z:z>9*0=1=4-3+9=1|1-4-4+1*0+a$123456789abcdefghijk', - [7, 10], false, [14, 15]); - testCharacterRangeFollow(2, 'Z:bc<6|x=b4|2-6$', [400, 407], false, [400, 401]); - testCharacterRangeFollow(3, 'Z:4>0-3+3$abc', [0, 3], false, [3, 3]); - testCharacterRangeFollow(4, 'Z:4>0-3+3$abc', [0, 3], true, [0, 0]); - testCharacterRangeFollow(5, 'Z:5>1+1=1-3+3$abcd', [1, 4], false, [5, 5]); - testCharacterRangeFollow(6, 'Z:5>1+1=1-3+3$abcd', [1, 4], true, [2, 2]); - testCharacterRangeFollow(7, 'Z:5>1+1=1-3+3$abcd', [0, 6], false, [1, 7]); - testCharacterRangeFollow(8, 'Z:5>1+1=1-3+3$abcd', [0, 3], false, [1, 2]); - testCharacterRangeFollow(9, 'Z:5>1+1=1-3+3$abcd', [2, 5], false, [5, 6]); - testCharacterRangeFollow(10, 'Z:2>1+1$a', [0, 0], false, [1, 1]); - testCharacterRangeFollow(11, 'Z:2>1+1$a', [0, 0], true, [0, 0]); - - it('opAttributeValue', async function () { - const p = new AttributePool(); - p.putAttrib(['name', 'david']); - p.putAttrib(['color', 'green']); - - const stringOp = (str) => Changeset.deserializeOps(str).next().value; - - expect(Changeset.opAttributeValue(stringOp('*0*1+1'), 'name', p)).to.equal('david'); - expect(Changeset.opAttributeValue(stringOp('*0+1'), 'name', p)).to.equal('david'); - expect(Changeset.opAttributeValue(stringOp('*1+1'), 'name', p)).to.equal(''); - expect(Changeset.opAttributeValue(stringOp('+1'), 'name', p)).to.equal(''); - expect(Changeset.opAttributeValue(stringOp('*0*1+1'), 'color', p)).to.equal('green'); - expect(Changeset.opAttributeValue(stringOp('*1+1'), 'color', p)).to.equal('green'); - expect(Changeset.opAttributeValue(stringOp('*0+1'), 'color', p)).to.equal(''); - expect(Changeset.opAttributeValue(stringOp('+1'), 'color', p)).to.equal(''); - }); - - const testAppendATextToAssembler = (testId, atext, correctOps) => { - it(`testAppendATextToAssembler#${testId}`, async function () { - const assem = Changeset.smartOpAssembler(); - for (const op of Changeset.opsFromAText(atext)) assem.append(op); - expect(assem.toString()).to.equal(correctOps); - }); - }; - - testAppendATextToAssembler(1, { - text: '\n', - attribs: '|1+1', - }, ''); - testAppendATextToAssembler(2, { - text: '\n\n', - attribs: '|2+2', - }, '|1+1'); - testAppendATextToAssembler(3, { - text: '\n\n', - attribs: '*x|2+2', - }, '*x|1+1'); - testAppendATextToAssembler(4, { - text: '\n\n', - attribs: '*x|1+1|1+1', - }, '*x|1+1'); - testAppendATextToAssembler(5, { - text: 'foo\n', - attribs: '|1+4', - }, '+3'); - testAppendATextToAssembler(6, { - text: '\nfoo\n', - attribs: '|2+5', - }, '|1+1+3'); - testAppendATextToAssembler(7, { - text: '\nfoo\n', - attribs: '*x|2+5', - }, '*x|1+1*x+3'); - testAppendATextToAssembler(8, { - text: '\n\n\nfoo\n', - attribs: '|2+2*x|2+5', - }, '|2+2*x|1+1*x+3'); - - const testMakeAttribsString = (testId, pool, opcode, attribs, correctString) => { - it(`testMakeAttribsString#${testId}`, async function () { - const p = poolOrArray(pool); - const str = Changeset.makeAttribsString(opcode, attribs, p); - expect(str).to.equal(correctString); - }); - }; - - testMakeAttribsString(1, ['bold,'], '+', [ - ['bold', ''], - ], ''); - testMakeAttribsString(2, ['abc,def', 'bold,'], '=', [ - ['bold', ''], - ], '*1'); - testMakeAttribsString(3, ['abc,def', 'bold,true'], '+', [ - ['abc', 'def'], - ['bold', 'true'], - ], '*0*1'); - testMakeAttribsString(4, ['abc,def', 'bold,true'], '+', [ - ['bold', 'true'], - ['abc', 'def'], - ], '*0*1'); - +describe('easysync-subAttribution', function () { const testSubattribution = (testId, astr, start, end, correctOutput) => { it(`subattribution#${testId}`, async function () { const str = Changeset.subattribution(astr, start, end); @@ -853,70 +959,4 @@ describe('easysync', function () { testSubattribution(40, '*0+2+1*1|1+3', 1, 5, '*0+1+1*1+2'); testSubattribution(41, '*0+2+1*1|1+3', 2, 6, '+1*1|1+3'); testSubattribution(42, '*0+2+1*1+3', 2, 6, '+1*1+3'); - - const testFilterAttribNumbers = (testId, cs, filter, correctOutput) => { - it(`testFilterAttribNumbers#${testId}`, async function () { - const str = Changeset.filterAttribNumbers(cs, filter); - expect(str).to.equal(correctOutput); - }); - }; - - testFilterAttribNumbers(1, '*0*1+1+2+3*1+4*2+5*0*2*1*b*c+6', - (n) => (n % 2) === 0, '*0+1+2+3+4*2+5*0*2*c+6'); - testFilterAttribNumbers(2, '*0*1+1+2+3*1+4*2+5*0*2*1*b*c+6', - (n) => (n % 2) === 1, '*1+1+2+3*1+4+5*1*b+6'); - - const testInverse = (testId, cs, lines, alines, pool, correctOutput) => { - it(`testInverse#${testId}`, async function () { - pool = poolOrArray(pool); - const str = Changeset.inverse(Changeset.checkRep(cs), lines, alines, pool); - expect(str).to.equal(correctOutput); - }); - }; - - // take "FFFFTTTTT" and apply "-FT--FFTT", the inverse of which is "--F--TT--" - testInverse(1, 'Z:9>0=1*0=1*1=1=2*0=2*1|1=2$', null, - ['+4*1+5'], ['bold,', 'bold,true'], 'Z:9>0=2*0=1=2*1=2$'); - - const testMutateTextLines = (testId, cs, lines, correctLines) => { - it(`testMutateTextLines#${testId}`, async function () { - const a = lines.slice(); - Changeset.mutateTextLines(cs, a); - expect(a).to.eql(correctLines); - }); - }; - - testMutateTextLines(1, 'Z:4<1|1-2-1|1+1+1$\nc', ['a\n', 'b\n'], ['\n', 'c\n']); - testMutateTextLines(2, 'Z:4>0|1-2-1|2+3$\nc\n', ['a\n', 'b\n'], ['\n', 'c\n', '\n']); - - const testInverseRandom = (randomSeed) => { - it(`testInverseRandom#${randomSeed}`, async function () { - const p = poolOrArray(['apple,', 'apple,true', 'banana,', 'banana,true']); - - const startText = `${randomMultiline(10, 20)}\n`; - const alines = - Changeset.splitAttributionLines(Changeset.makeAttribution(startText), startText); - const lines = startText.slice(0, -1).split('\n').map((s) => `${s}\n`); - - const stylifier = randomTestChangeset(startText, true)[0]; - - Changeset.mutateAttributionLines(stylifier, alines, p); - Changeset.mutateTextLines(stylifier, lines); - - const changeset = randomTestChangeset(lines.join(''), true)[0]; - const inverseChangeset = Changeset.inverse(changeset, lines, alines, p); - - const origLines = lines.slice(); - const origALines = alines.slice(); - - Changeset.mutateTextLines(changeset, lines); - Changeset.mutateAttributionLines(changeset, alines, p); - Changeset.mutateTextLines(inverseChangeset, lines); - Changeset.mutateAttributionLines(inverseChangeset, alines, p); - expect(lines).to.eql(origLines); - expect(alines).to.eql(origALines); - }); - }; - - for (let i = 0; i < 30; i++) testInverseRandom(i); }); From 0983985dd555c7596aaab3e994ffefda937c16e1 Mon Sep 17 00:00:00 2001 From: webzwo0i Date: Sun, 17 Oct 2021 23:57:47 +0200 Subject: [PATCH 034/446] easysync tests: Split into multiple files --- src/tests/frontend/easysync-helper.js | 222 ++++ .../frontend/specs/easysync-assembler.js | 63 ++ src/tests/frontend/specs/easysync-compose.js | 53 + src/tests/frontend/specs/easysync-follow.js | 82 ++ .../frontend/specs/easysync-inverseRandom.js | 53 + .../frontend/specs/easysync-mutations.js | 304 ++++++ src/tests/frontend/specs/easysync-other.js | 141 +++ .../frontend/specs/easysync-subAttribution.js | 55 + src/tests/frontend/specs/easysync.js | 962 ------------------ 9 files changed, 973 insertions(+), 962 deletions(-) create mode 100644 src/tests/frontend/easysync-helper.js create mode 100644 src/tests/frontend/specs/easysync-assembler.js create mode 100644 src/tests/frontend/specs/easysync-compose.js create mode 100644 src/tests/frontend/specs/easysync-follow.js create mode 100644 src/tests/frontend/specs/easysync-inverseRandom.js create mode 100644 src/tests/frontend/specs/easysync-mutations.js create mode 100644 src/tests/frontend/specs/easysync-other.js create mode 100644 src/tests/frontend/specs/easysync-subAttribution.js delete mode 100644 src/tests/frontend/specs/easysync.js diff --git a/src/tests/frontend/easysync-helper.js b/src/tests/frontend/easysync-helper.js new file mode 100644 index 000000000..6b4bfc959 --- /dev/null +++ b/src/tests/frontend/easysync-helper.js @@ -0,0 +1,222 @@ +'use strict'; + +const Changeset = require('../../static/js/Changeset'); +const AttributePool = require('../../static/js/AttributePool'); + +const randInt = (maxValue) => Math.floor(Math.random() * maxValue); + +const poolOrArray = (attribs) => { + if (attribs.getAttrib) { + return attribs; // it's already an attrib pool + } else { + // assume it's an array of attrib strings to be split and added + const p = new AttributePool(); + attribs.forEach((kv) => { + p.putAttrib(kv.split(',')); + }); + return p; + } +}; +exports.poolOrArray = poolOrArray; + +const randomInlineString = (len) => { + const assem = Changeset.stringAssembler(); + for (let i = 0; i < len; i++) { + assem.append(String.fromCharCode(randInt(26) + 97)); + } + return assem.toString(); +}; + +const randomMultiline = (approxMaxLines, approxMaxCols) => { + const numParts = randInt(approxMaxLines * 2) + 1; + const txt = Changeset.stringAssembler(); + txt.append(randInt(2) ? '\n' : ''); + for (let i = 0; i < numParts; i++) { + if ((i % 2) === 0) { + if (randInt(10)) { + txt.append(randomInlineString(randInt(approxMaxCols) + 1)); + } else { + txt.append('\n'); + } + } else { + txt.append('\n'); + } + } + return txt.toString(); +}; +exports.randomMultiline = randomMultiline; + +const randomStringOperation = (numCharsLeft) => { + let result; + switch (randInt(9)) { + case 0: + { + // insert char + result = { + insert: randomInlineString(1), + }; + break; + } + case 1: + { + // delete char + result = { + remove: 1, + }; + break; + } + case 2: + { + // skip char + result = { + skip: 1, + }; + break; + } + case 3: + { + // insert small + result = { + insert: randomInlineString(randInt(4) + 1), + }; + break; + } + case 4: + { + // delete small + result = { + remove: randInt(4) + 1, + }; + break; + } + case 5: + { + // skip small + result = { + skip: randInt(4) + 1, + }; + break; + } + case 6: + { + // insert multiline; + result = { + insert: randomMultiline(5, 20), + }; + break; + } + case 7: + { + // delete multiline + result = { + remove: Math.round(numCharsLeft * Math.random() * Math.random()), + }; + break; + } + case 8: + { + // skip multiline + result = { + skip: Math.round(numCharsLeft * Math.random() * Math.random()), + }; + break; + } + case 9: + { + // delete to end + result = { + remove: numCharsLeft, + }; + break; + } + case 10: + { + // skip to end + result = { + skip: numCharsLeft, + }; + break; + } + } + const maxOrig = numCharsLeft - 1; + if ('remove' in result) { + result.remove = Math.min(result.remove, maxOrig); + } else if ('skip' in result) { + result.skip = Math.min(result.skip, maxOrig); + } + return result; +}; + +const randomTwoPropAttribs = (opcode) => { + // assumes attrib pool like ['apple,','apple,true','banana,','banana,true'] + if (opcode === '-' || randInt(3)) { + return ''; + } else if (randInt(3)) { // eslint-disable-line no-dupe-else-if + if (opcode === '+' || randInt(2)) { + return `*${Changeset.numToString(randInt(2) * 2 + 1)}`; + } else { + return `*${Changeset.numToString(randInt(2) * 2)}`; + } + } else if (opcode === '+' || randInt(4) === 0) { + return '*1*3'; + } else { + return ['*0*2', '*0*3', '*1*2'][randInt(3)]; + } +}; + +const randomTestChangeset = (origText, withAttribs) => { + const charBank = Changeset.stringAssembler(); + let textLeft = origText; // always keep final newline + const outTextAssem = Changeset.stringAssembler(); + const opAssem = Changeset.smartOpAssembler(); + const oldLen = origText.length; + + const nextOp = new Changeset.Op(); + + const appendMultilineOp = (opcode, txt) => { + nextOp.opcode = opcode; + if (withAttribs) { + nextOp.attribs = randomTwoPropAttribs(opcode); + } + txt.replace(/\n|[^\n]+/g, (t) => { + if (t === '\n') { + nextOp.chars = 1; + nextOp.lines = 1; + opAssem.append(nextOp); + } else { + nextOp.chars = t.length; + nextOp.lines = 0; + opAssem.append(nextOp); + } + return ''; + }); + }; + + const doOp = () => { + const o = randomStringOperation(textLeft.length); + if (o.insert) { + const txt = o.insert; + charBank.append(txt); + outTextAssem.append(txt); + appendMultilineOp('+', txt); + } else if (o.skip) { + const txt = textLeft.substring(0, o.skip); + textLeft = textLeft.substring(o.skip); + outTextAssem.append(txt); + appendMultilineOp('=', txt); + } else if (o.remove) { + const txt = textLeft.substring(0, o.remove); + textLeft = textLeft.substring(o.remove); + appendMultilineOp('-', txt); + } + }; + + while (textLeft.length > 1) doOp(); + for (let i = 0; i < 5; i++) doOp(); // do some more (only insertions will happen) + const outText = `${outTextAssem.toString()}\n`; + opAssem.endDocument(); + const cs = Changeset.pack(oldLen, outText.length, opAssem.toString(), charBank.toString()); + Changeset.checkRep(cs); + return [cs, outText]; +}; +exports.randomTestChangeset = randomTestChangeset; diff --git a/src/tests/frontend/specs/easysync-assembler.js b/src/tests/frontend/specs/easysync-assembler.js new file mode 100644 index 000000000..d9ce04ae2 --- /dev/null +++ b/src/tests/frontend/specs/easysync-assembler.js @@ -0,0 +1,63 @@ +'use strict'; + +const Changeset = require('../../../static/js/Changeset'); + +describe('easysync-assembler', function () { + it('opAssembler', async function () { + const x = '-c*3*4+6|3=az*asdf0*1*2*3+1=1-1+1*0+1=1-1+1|c=c-1'; + const assem = Changeset.opAssembler(); + for (const op of Changeset.deserializeOps(x)) assem.append(op); + expect(assem.toString()).to.equal(x); + }); + + it('smartOpAssembler', async function () { + const x = '-c*3*4+6|3=az*asdf0*1*2*3+1=1-1+1*0+1=1-1+1|c=c-1'; + const assem = Changeset.smartOpAssembler(); + for (const op of Changeset.deserializeOps(x)) assem.append(op); + assem.endDocument(); + expect(assem.toString()).to.equal(x); + }); + + describe('append atext to assembler', function () { + const testAppendATextToAssembler = (testId, atext, correctOps) => { + it(`testAppendATextToAssembler#${testId}`, async function () { + const assem = Changeset.smartOpAssembler(); + for (const op of Changeset.opsFromAText(atext)) assem.append(op); + expect(assem.toString()).to.equal(correctOps); + }); + }; + + testAppendATextToAssembler(1, { + text: '\n', + attribs: '|1+1', + }, ''); + testAppendATextToAssembler(2, { + text: '\n\n', + attribs: '|2+2', + }, '|1+1'); + testAppendATextToAssembler(3, { + text: '\n\n', + attribs: '*x|2+2', + }, '*x|1+1'); + testAppendATextToAssembler(4, { + text: '\n\n', + attribs: '*x|1+1|1+1', + }, '*x|1+1'); + testAppendATextToAssembler(5, { + text: 'foo\n', + attribs: '|1+4', + }, '+3'); + testAppendATextToAssembler(6, { + text: '\nfoo\n', + attribs: '|2+5', + }, '|1+1+3'); + testAppendATextToAssembler(7, { + text: '\nfoo\n', + attribs: '*x|2+5', + }, '*x|1+1*x+3'); + testAppendATextToAssembler(8, { + text: '\n\n\nfoo\n', + attribs: '|2+2*x|2+5', + }, '|2+2*x|1+1*x+3'); + }); +}); diff --git a/src/tests/frontend/specs/easysync-compose.js b/src/tests/frontend/specs/easysync-compose.js new file mode 100644 index 000000000..69757763c --- /dev/null +++ b/src/tests/frontend/specs/easysync-compose.js @@ -0,0 +1,53 @@ +'use strict'; + +const Changeset = require('../../../static/js/Changeset'); +const AttributePool = require('../../../static/js/AttributePool'); +const {randomMultiline, randomTestChangeset} = require('../easysync-helper.js'); + +describe('easysync-compose', function () { + describe('compose', function () { + const testCompose = (randomSeed) => { + it(`testCompose#${randomSeed}`, async function () { + const p = new AttributePool(); + + const startText = `${randomMultiline(10, 20)}\n`; + + const x1 = randomTestChangeset(startText); + const change1 = x1[0]; + const text1 = x1[1]; + + const x2 = randomTestChangeset(text1); + const change2 = x2[0]; + const text2 = x2[1]; + + const x3 = randomTestChangeset(text2); + const change3 = x3[0]; + const text3 = x3[1]; + + const change12 = Changeset.checkRep(Changeset.compose(change1, change2, p)); + const change23 = Changeset.checkRep(Changeset.compose(change2, change3, p)); + const change123 = Changeset.checkRep(Changeset.compose(change12, change3, p)); + const change123a = Changeset.checkRep(Changeset.compose(change1, change23, p)); + expect(change123a).to.equal(change123); + + expect(Changeset.applyToText(change12, startText)).to.equal(text2); + expect(Changeset.applyToText(change23, text1)).to.equal(text3); + expect(Changeset.applyToText(change123, startText)).to.equal(text3); + }); + }; + + for (let i = 0; i < 30; i++) testCompose(i); + }); + + describe('compose attributes', function () { + it('simpleComposeAttributesTest', async function () { + const p = new AttributePool(); + p.putAttrib(['bold', '']); + p.putAttrib(['bold', 'true']); + const cs1 = Changeset.checkRep('Z:2>1*1+1*1=1$x'); + const cs2 = Changeset.checkRep('Z:3>0*0|1=3$'); + const cs12 = Changeset.checkRep(Changeset.compose(cs1, cs2, p)); + expect(cs12).to.equal('Z:2>1+1*0|1=2$x'); + }); + }); +}); diff --git a/src/tests/frontend/specs/easysync-follow.js b/src/tests/frontend/specs/easysync-follow.js new file mode 100644 index 000000000..9ec5a7e83 --- /dev/null +++ b/src/tests/frontend/specs/easysync-follow.js @@ -0,0 +1,82 @@ +'use strict'; + +const Changeset = require('../../../static/js/Changeset'); +const AttributePool = require('../../../static/js/AttributePool'); +const {randomMultiline, randomTestChangeset} = require('../easysync-helper.js'); + +describe('easysync-follow', function () { + describe('follow & compose', function () { + const testFollow = (randomSeed) => { + it(`testFollow#${randomSeed}`, async function () { + const p = new AttributePool(); + + const startText = `${randomMultiline(10, 20)}\n`; + + const cs1 = randomTestChangeset(startText)[0]; + const cs2 = randomTestChangeset(startText)[0]; + + const afb = Changeset.checkRep(Changeset.follow(cs1, cs2, false, p)); + const bfa = Changeset.checkRep(Changeset.follow(cs2, cs1, true, p)); + + const merge1 = Changeset.checkRep(Changeset.compose(cs1, afb)); + const merge2 = Changeset.checkRep(Changeset.compose(cs2, bfa)); + + expect(merge2).to.equal(merge1); + }); + }; + + for (let i = 0; i < 30; i++) testFollow(i); + }); + + describe('followAttributes & composeAttributes', function () { + const p = new AttributePool(); + p.putAttrib(['x', '']); + p.putAttrib(['x', 'abc']); + p.putAttrib(['x', 'def']); + p.putAttrib(['y', '']); + p.putAttrib(['y', 'abc']); + p.putAttrib(['y', 'def']); + let n = 0; + + const testFollow = (a, b, afb, bfa, merge) => { + it(`manual #${++n}`, async function () { + expect(Changeset.exportedForTestingOnly.followAttributes(a, b, p)).to.equal(afb); + expect(Changeset.exportedForTestingOnly.followAttributes(b, a, p)).to.equal(bfa); + expect(Changeset.composeAttributes(a, afb, true, p)).to.equal(merge); + expect(Changeset.composeAttributes(b, bfa, true, p)).to.equal(merge); + }); + }; + + testFollow('', '', '', '', ''); + testFollow('*0', '', '', '*0', '*0'); + testFollow('*0', '*0', '', '', '*0'); + testFollow('*0', '*1', '', '*0', '*0'); + testFollow('*1', '*2', '', '*1', '*1'); + testFollow('*0*1', '', '', '*0*1', '*0*1'); + testFollow('*0*4', '*2*3', '*3', '*0', '*0*3'); + testFollow('*0*4', '*2', '', '*0*4', '*0*4'); + }); + + describe('chracterRangeFollow', function () { + const testCharacterRangeFollow = (testId, cs, oldRange, insertionsAfter, correctNewRange) => { + it(`testCharacterRangeFollow#${testId}`, async function () { + cs = Changeset.checkRep(cs); + expect(Changeset.characterRangeFollow(cs, oldRange[0], oldRange[1], insertionsAfter)) + .to.eql(correctNewRange); + }); + }; + + testCharacterRangeFollow(1, 'Z:z>9*0=1=4-3+9=1|1-4-4+1*0+a$123456789abcdefghijk', + [7, 10], false, [14, 15]); + testCharacterRangeFollow(2, 'Z:bc<6|x=b4|2-6$', [400, 407], false, [400, 401]); + testCharacterRangeFollow(3, 'Z:4>0-3+3$abc', [0, 3], false, [3, 3]); + testCharacterRangeFollow(4, 'Z:4>0-3+3$abc', [0, 3], true, [0, 0]); + testCharacterRangeFollow(5, 'Z:5>1+1=1-3+3$abcd', [1, 4], false, [5, 5]); + testCharacterRangeFollow(6, 'Z:5>1+1=1-3+3$abcd', [1, 4], true, [2, 2]); + testCharacterRangeFollow(7, 'Z:5>1+1=1-3+3$abcd', [0, 6], false, [1, 7]); + testCharacterRangeFollow(8, 'Z:5>1+1=1-3+3$abcd', [0, 3], false, [1, 2]); + testCharacterRangeFollow(9, 'Z:5>1+1=1-3+3$abcd', [2, 5], false, [5, 6]); + testCharacterRangeFollow(10, 'Z:2>1+1$a', [0, 0], false, [1, 1]); + testCharacterRangeFollow(11, 'Z:2>1+1$a', [0, 0], true, [0, 0]); + }); +}); diff --git a/src/tests/frontend/specs/easysync-inverseRandom.js b/src/tests/frontend/specs/easysync-inverseRandom.js new file mode 100644 index 000000000..41ef86d57 --- /dev/null +++ b/src/tests/frontend/specs/easysync-inverseRandom.js @@ -0,0 +1,53 @@ +'use strict'; + +const Changeset = require('../../../static/js/Changeset'); +const {randomMultiline, randomTestChangeset, poolOrArray} = require('../easysync-helper.js'); + +describe('easysync-inverseRandom', function () { + describe('inverse random', function () { + const testInverseRandom = (randomSeed) => { + it(`testInverseRandom#${randomSeed}`, async function () { + const p = poolOrArray(['apple,', 'apple,true', 'banana,', 'banana,true']); + + const startText = `${randomMultiline(10, 20)}\n`; + const alines = + Changeset.splitAttributionLines(Changeset.makeAttribution(startText), startText); + const lines = startText.slice(0, -1).split('\n').map((s) => `${s}\n`); + + const stylifier = randomTestChangeset(startText, true)[0]; + + Changeset.mutateAttributionLines(stylifier, alines, p); + Changeset.mutateTextLines(stylifier, lines); + + const changeset = randomTestChangeset(lines.join(''), true)[0]; + const inverseChangeset = Changeset.inverse(changeset, lines, alines, p); + + const origLines = lines.slice(); + const origALines = alines.slice(); + + Changeset.mutateTextLines(changeset, lines); + Changeset.mutateAttributionLines(changeset, alines, p); + Changeset.mutateTextLines(inverseChangeset, lines); + Changeset.mutateAttributionLines(inverseChangeset, alines, p); + expect(lines).to.eql(origLines); + expect(alines).to.eql(origALines); + }); + }; + + for (let i = 0; i < 30; i++) testInverseRandom(i); + }); + + describe('inverse', function () { + const testInverse = (testId, cs, lines, alines, pool, correctOutput) => { + it(`testInverse#${testId}`, async function () { + pool = poolOrArray(pool); + const str = Changeset.inverse(Changeset.checkRep(cs), lines, alines, pool); + expect(str).to.equal(correctOutput); + }); + }; + + // take "FFFFTTTTT" and apply "-FT--FFTT", the inverse of which is "--F--TT--" + testInverse(1, 'Z:9>0=1*0=1*1=1=2*0=2*1|1=2$', null, + ['+4*1+5'], ['bold,', 'bold,true'], 'Z:9>0=2*0=1=2*1=2$'); + }); +}); diff --git a/src/tests/frontend/specs/easysync-mutations.js b/src/tests/frontend/specs/easysync-mutations.js new file mode 100644 index 000000000..7cf43c8b7 --- /dev/null +++ b/src/tests/frontend/specs/easysync-mutations.js @@ -0,0 +1,304 @@ +'use strict'; + +const Changeset = require('../../../static/js/Changeset'); +const AttributePool = require('../../../static/js/AttributePool'); +const {poolOrArray} = require('../easysync-helper.js'); + +describe('easysync-mutations', function () { + const applyMutations = (mu, arrayOfArrays) => { + arrayOfArrays.forEach((a) => { + const result = mu[a[0]](...a.slice(1)); + if (a[0] === 'remove' && a[3]) { + expect(result).to.equal(a[3]); + } + }); + }; + + const mutationsToChangeset = (oldLen, arrayOfArrays) => { + const assem = Changeset.smartOpAssembler(); + const op = new Changeset.Op(); + const bank = Changeset.stringAssembler(); + let oldPos = 0; + let newLen = 0; + arrayOfArrays.forEach((a) => { + if (a[0] === 'skip') { + op.opcode = '='; + op.chars = a[1]; + op.lines = (a[2] || 0); + assem.append(op); + oldPos += op.chars; + newLen += op.chars; + } else if (a[0] === 'remove') { + op.opcode = '-'; + op.chars = a[1]; + op.lines = (a[2] || 0); + assem.append(op); + oldPos += op.chars; + } else if (a[0] === 'insert') { + op.opcode = '+'; + bank.append(a[1]); + op.chars = a[1].length; + op.lines = (a[2] || 0); + assem.append(op); + newLen += op.chars; + } + }); + newLen += oldLen - oldPos; + assem.endDocument(); + return Changeset.pack(oldLen, newLen, assem.toString(), bank.toString()); + }; + + const runMutationTest = (testId, origLines, muts, correct) => { + it(`runMutationTest#${testId}`, async function () { + let lines = origLines.slice(); + const mu = new Changeset.exportedForTestingOnly.TextLinesMutator(lines); + applyMutations(mu, muts); + mu.close(); + expect(lines).to.eql(correct); + + const inText = origLines.join(''); + const cs = mutationsToChangeset(inText.length, muts); + lines = origLines.slice(); + Changeset.mutateTextLines(cs, lines); + expect(lines).to.eql(correct); + + const correctText = correct.join(''); + const outText = Changeset.applyToText(cs, inText); + expect(outText).to.equal(correctText); + }); + }; + + runMutationTest(1, ['apple\n', 'banana\n', 'cabbage\n', 'duffle\n', 'eggplant\n'], [ + ['remove', 1, 0, 'a'], + ['insert', 'tu'], + ['remove', 1, 0, 'p'], + ['skip', 4, 1], + ['skip', 7, 1], + ['insert', 'cream\npie\n', 2], + ['skip', 2], + ['insert', 'bot'], + ['insert', '\n', 1], + ['insert', 'bu'], + ['skip', 3], + ['remove', 3, 1, 'ge\n'], + ['remove', 6, 0, 'duffle'], + ], ['tuple\n', 'banana\n', 'cream\n', 'pie\n', 'cabot\n', 'bubba\n', 'eggplant\n']); + + runMutationTest(2, ['apple\n', 'banana\n', 'cabbage\n', 'duffle\n', 'eggplant\n'], [ + ['remove', 1, 0, 'a'], + ['remove', 1, 0, 'p'], + ['insert', 'tu'], + ['skip', 11, 2], + ['insert', 'cream\npie\n', 2], + ['skip', 2], + ['insert', 'bot'], + ['insert', '\n', 1], + ['insert', 'bu'], + ['skip', 3], + ['remove', 3, 1, 'ge\n'], + ['remove', 6, 0, 'duffle'], + ], ['tuple\n', 'banana\n', 'cream\n', 'pie\n', 'cabot\n', 'bubba\n', 'eggplant\n']); + + runMutationTest(3, ['apple\n', 'banana\n', 'cabbage\n', 'duffle\n', 'eggplant\n'], [ + ['remove', 6, 1, 'apple\n'], + ['skip', 15, 2], + ['skip', 6], + ['remove', 1, 1, '\n'], + ['remove', 8, 0, 'eggplant'], + ['skip', 1, 1], + ], ['banana\n', 'cabbage\n', 'duffle\n']); + + runMutationTest(4, ['15\n'], [ + ['skip', 1], + ['insert', '\n2\n3\n4\n', 4], + ['skip', 2, 1], + ], ['1\n', '2\n', '3\n', '4\n', '5\n']); + + runMutationTest(5, ['1\n', '2\n', '3\n', '4\n', '5\n'], [ + ['skip', 1], + ['remove', 7, 4, '\n2\n3\n4\n'], + ['skip', 2, 1], + ], ['15\n']); + + runMutationTest(6, ['123\n', 'abc\n', 'def\n', 'ghi\n', 'xyz\n'], [ + ['insert', '0'], + ['skip', 4, 1], + ['skip', 4, 1], + ['remove', 8, 2, 'def\nghi\n'], + ['skip', 4, 1], + ], ['0123\n', 'abc\n', 'xyz\n']); + + runMutationTest(7, ['apple\n', 'banana\n', 'cabbage\n', 'duffle\n', 'eggplant\n'], [ + ['remove', 6, 1, 'apple\n'], + ['skip', 15, 2, true], + ['skip', 6, 0, true], + ['remove', 1, 1, '\n'], + ['remove', 8, 0, 'eggplant'], + ['skip', 1, 1, true], + ], ['banana\n', 'cabbage\n', 'duffle\n']); + + it('mutatorHasMore', async function () { + const lines = ['1\n', '2\n', '3\n', '4\n']; + let mu; + + mu = new Changeset.exportedForTestingOnly.TextLinesMutator(lines); + expect(mu.hasMore()).to.be(true); + mu.skip(8, 4); + expect(mu.hasMore()).to.be(false); + mu.close(); + expect(mu.hasMore()).to.be(false); + + // still 1,2,3,4 + mu = new Changeset.exportedForTestingOnly.TextLinesMutator(lines); + expect(mu.hasMore()).to.be(true); + mu.remove(2, 1); + expect(mu.hasMore()).to.be(true); + mu.skip(2, 1); + expect(mu.hasMore()).to.be(true); + mu.skip(2, 1); + expect(mu.hasMore()).to.be(true); + mu.skip(2, 1); + expect(mu.hasMore()).to.be(false); + mu.insert('5\n', 1); + expect(mu.hasMore()).to.be(false); + mu.close(); + expect(mu.hasMore()).to.be(false); + + // 2,3,4,5 now + mu = new Changeset.exportedForTestingOnly.TextLinesMutator(lines); + expect(mu.hasMore()).to.be(true); + mu.remove(6, 3); + expect(mu.hasMore()).to.be(true); + mu.remove(2, 1); + expect(mu.hasMore()).to.be(false); + mu.insert('hello\n', 1); + expect(mu.hasMore()).to.be(false); + mu.close(); + expect(mu.hasMore()).to.be(false); + }); + + describe('mutateTextLines', function () { + const testMutateTextLines = (testId, cs, lines, correctLines) => { + it(`testMutateTextLines#${testId}`, async function () { + const a = lines.slice(); + Changeset.mutateTextLines(cs, a); + expect(a).to.eql(correctLines); + }); + }; + + testMutateTextLines(1, 'Z:4<1|1-2-1|1+1+1$\nc', ['a\n', 'b\n'], ['\n', 'c\n']); + testMutateTextLines(2, 'Z:4>0|1-2-1|2+3$\nc\n', ['a\n', 'b\n'], ['\n', 'c\n', '\n']); + }); + + describe('mutate attributions', function () { + const testPoolWithChars = (() => { + const p = new AttributePool(); + p.putAttrib(['char', 'newline']); + for (let i = 1; i < 36; i++) { + p.putAttrib(['char', Changeset.numToString(i)]); + } + p.putAttrib(['char', '']); + return p; + })(); + + const runMutateAttributionTest = (testId, attribs, cs, alines, outCorrect) => { + it(`runMutateAttributionTest#${testId}`, async function () { + const p = poolOrArray(attribs); + const alines2 = Array.prototype.slice.call(alines); + Changeset.mutateAttributionLines(Changeset.checkRep(cs), alines2, p); + expect(alines2).to.eql(outCorrect); + + const removeQuestionMarks = (a) => a.replace(/\?/g, ''); + const inMerged = Changeset.joinAttributionLines(alines.map(removeQuestionMarks)); + const correctMerged = Changeset.joinAttributionLines(outCorrect.map(removeQuestionMarks)); + const mergedResult = Changeset.applyToAttribution(cs, inMerged, p); + expect(mergedResult).to.equal(correctMerged); + }); + }; + + // turn 123\n 456\n 789\n into 123\n 456\n 789\n + runMutateAttributionTest(1, + ['bold,true'], 'Z:c>0|1=4=1*0=1$', ['|1+4', '|1+4', '|1+4'], ['|1+4', '+1*0+1|1+2', '|1+4'] + ); + + // make a document bold + runMutateAttributionTest(2, + ['bold,true'], 'Z:c>0*0|3=c$', ['|1+4', '|1+4', '|1+4'], ['*0|1+4', '*0|1+4', '*0|1+4']); + + // clear bold on document + runMutateAttributionTest(3, + ['bold,', 'bold,true'], 'Z:c>0*0|3=c$', + ['*1+1+1*1+1|1+1', '+1*1+1|1+2', '*1+1+1*1+1|1+1'], ['|1+4', '|1+4', '|1+4']); + + // add a character on line 3 of a document with 5 blank lines, and make sure + // the optimization that skips purely-kept lines is working; if any attribution string + // with a '?' is parsed it will cause an error. + runMutateAttributionTest(4, + ['foo,bar', 'line,1', 'line,2', 'line,3', 'line,4', 'line,5'], + 'Z:5>1|2=2+1$x', ['?*1|1+1', '?*2|1+1', '*3|1+1', '?*4|1+1', '?*5|1+1'], + ['?*1|1+1', '?*2|1+1', '+1*3|1+1', '?*4|1+1', '?*5|1+1']); + + // based on runMutationTest#1 + runMutateAttributionTest(5, testPoolWithChars, + 'Z:11>7-2*t+1*u+1|2=b|2+a=2*b+1*o+1*t+1*0|1+1*b+1*u+1=3|1-3-6$tucream\npie\nbot\nbu', + [ + '*a+1*p+2*l+1*e+1*0|1+1', + '*b+1*a+1*n+1*a+1*n+1*a+1*0|1+1', + '*c+1*a+1*b+2*a+1*g+1*e+1*0|1+1', + '*d+1*u+1*f+2*l+1*e+1*0|1+1', + '*e+1*g+2*p+1*l+1*a+1*n+1*t+1*0|1+1', + ], + [ + '*t+1*u+1*p+1*l+1*e+1*0|1+1', + '*b+1*a+1*n+1*a+1*n+1*a+1*0|1+1', + '|1+6', + '|1+4', + '*c+1*a+1*b+1*o+1*t+1*0|1+1', + '*b+1*u+1*b+2*a+1*0|1+1', + '*e+1*g+2*p+1*l+1*a+1*n+1*t+1*0|1+1', + ]); + + // based on runMutationTest#3 + runMutateAttributionTest(6, testPoolWithChars, + 'Z:117=1|4+7$\n2\n3\n4\n', + ['*1+1*5|1+2'], ['*1+1|1+1', '|1+2', '|1+2', '|1+2', '*5|1+2']); + + // based on runMutationTest#5 + runMutateAttributionTest(8, testPoolWithChars, 'Z:a<7=1|4-7$', + ['*1|1+2', '*2|1+2', '*3|1+2', '*4|1+2', '*5|1+2'], ['*1+1*5|1+2']); + + // based on runMutationTest#6 + runMutateAttributionTest(9, testPoolWithChars, 'Z:k<7*0+1*10|2=8|2-8$0', + [ + '*1+1*2+1*3+1|1+1', + '*a+1*b+1*c+1|1+1', + '*d+1*e+1*f+1|1+1', + '*g+1*h+1*i+1|1+1', + '?*x+1*y+1*z+1|1+1', + ], + ['*0+1|1+4', '|1+4', '?*x+1*y+1*z+1|1+1']); + + runMutateAttributionTest(10, testPoolWithChars, 'Z:6>4=1+1=1+1|1=1+1=1*0+1$abcd', + ['|1+3', '|1+3'], ['|1+5', '+2*0+1|1+2']); + + + runMutateAttributionTest(11, testPoolWithChars, 'Z:s>1|1=4=6|1+1$\n', + ['*0|1+4', '*0|1+8', '*0+5|1+1', '*0|1+1', '*0|1+5', '*0|1+1', '*0|1+1', '*0|1+1', '|1+1'], + [ + '*0|1+4', + '*0+6|1+1', + '*0|1+2', + '*0+5|1+1', + '*0|1+1', + '*0|1+5', + '*0|1+1', + '*0|1+1', + '*0|1+1', + '|1+1', + ]); + }); +}); diff --git a/src/tests/frontend/specs/easysync-other.js b/src/tests/frontend/specs/easysync-other.js new file mode 100644 index 000000000..26376713a --- /dev/null +++ b/src/tests/frontend/specs/easysync-other.js @@ -0,0 +1,141 @@ +'use strict'; + +const Changeset = require('../../../static/js/Changeset'); +const AttributePool = require('../../../static/js/AttributePool'); +const {randomMultiline, poolOrArray} = require('../easysync-helper.js'); + +describe('easysync-other', function () { + describe('filter attribute numbers', function () { + const testFilterAttribNumbers = (testId, cs, filter, correctOutput) => { + it(`testFilterAttribNumbers#${testId}`, async function () { + const str = Changeset.filterAttribNumbers(cs, filter); + expect(str).to.equal(correctOutput); + }); + }; + + testFilterAttribNumbers(1, '*0*1+1+2+3*1+4*2+5*0*2*1*b*c+6', + (n) => (n % 2) === 0, '*0+1+2+3+4*2+5*0*2*c+6'); + testFilterAttribNumbers(2, '*0*1+1+2+3*1+4*2+5*0*2*1*b*c+6', + (n) => (n % 2) === 1, '*1+1+2+3*1+4+5*1*b+6'); + }); + + describe('make attribs string', function () { + const testMakeAttribsString = (testId, pool, opcode, attribs, correctString) => { + it(`testMakeAttribsString#${testId}`, async function () { + const p = poolOrArray(pool); + const str = Changeset.makeAttribsString(opcode, attribs, p); + expect(str).to.equal(correctString); + }); + }; + + testMakeAttribsString(1, ['bold,'], '+', [ + ['bold', ''], + ], ''); + testMakeAttribsString(2, ['abc,def', 'bold,'], '=', [ + ['bold', ''], + ], '*1'); + testMakeAttribsString(3, ['abc,def', 'bold,true'], '+', [ + ['abc', 'def'], + ['bold', 'true'], + ], '*0*1'); + testMakeAttribsString(4, ['abc,def', 'bold,true'], '+', [ + ['bold', 'true'], + ['abc', 'def'], + ], '*0*1'); + }); + + describe('other', function () { + it('testMoveOpsToNewPool', async function () { + const pool1 = new AttributePool(); + const pool2 = new AttributePool(); + + pool1.putAttrib(['baz', 'qux']); + pool1.putAttrib(['foo', 'bar']); + + pool2.putAttrib(['foo', 'bar']); + + expect(Changeset.moveOpsToNewPool('Z:1>2*1+1*0+1$ab', pool1, pool2)) + .to.equal('Z:1>2*0+1*1+1$ab'); + expect(Changeset.moveOpsToNewPool('*1+1*0+1', pool1, pool2)).to.equal('*0+1*1+1'); + }); + + it('testMakeSplice', async function () { + const t = 'a\nb\nc\n'; + const t2 = Changeset.applyToText(Changeset.makeSplice(t, 5, 0, 'def'), t); + expect(t2).to.equal('a\nb\ncdef\n'); + }); + + it('testToSplices', async function () { + const cs = Changeset.checkRep('Z:z>9*0=1=4-3+9=1|1-4-4+1*0+a$123456789abcdefghijk'); + const correctSplices = [ + [5, 8, '123456789'], + [9, 17, 'abcdefghijk'], + ]; + expect(Changeset.exportedForTestingOnly.toSplices(cs)).to.eql(correctSplices); + }); + + it('opAttributeValue', async function () { + const p = new AttributePool(); + p.putAttrib(['name', 'david']); + p.putAttrib(['color', 'green']); + + const stringOp = (str) => Changeset.deserializeOps(str).next().value; + + expect(Changeset.opAttributeValue(stringOp('*0*1+1'), 'name', p)).to.equal('david'); + expect(Changeset.opAttributeValue(stringOp('*0+1'), 'name', p)).to.equal('david'); + expect(Changeset.opAttributeValue(stringOp('*1+1'), 'name', p)).to.equal(''); + expect(Changeset.opAttributeValue(stringOp('+1'), 'name', p)).to.equal(''); + expect(Changeset.opAttributeValue(stringOp('*0*1+1'), 'color', p)).to.equal('green'); + expect(Changeset.opAttributeValue(stringOp('*1+1'), 'color', p)).to.equal('green'); + expect(Changeset.opAttributeValue(stringOp('*0+1'), 'color', p)).to.equal(''); + expect(Changeset.opAttributeValue(stringOp('+1'), 'color', p)).to.equal(''); + }); + + describe('applyToAttribution', function () { + const runApplyToAttributionTest = (testId, attribs, cs, inAttr, outCorrect) => { + it(`applyToAttribution#${testId}`, async function () { + const p = poolOrArray(attribs); + const result = Changeset.applyToAttribution(Changeset.checkRep(cs), inAttr, p); + expect(result).to.equal(outCorrect); + }); + }; + + // turn cactus\n into actusabcd\n + runApplyToAttributionTest(1, + ['bold,', 'bold,true'], 'Z:7>3-1*0=1*1=1=3+4$abcd', '+1*1+1|1+5', '+1*1+1|1+8'); + + // turn "david\ngreenspan\n" into "david\ngreen\n" + runApplyToAttributionTest(2, + ['bold,', 'bold,true'], 'Z:g<4*1|1=6*1=5-4$', '|2+g', '*1|1+6*1+5|1+1'); + }); + + describe('split/join attribution lines', function () { + const testSplitJoinAttributionLines = (randomSeed) => { + const stringToOps = (str) => { + const assem = Changeset.mergingOpAssembler(); + const o = new Changeset.Op('+'); + o.chars = 1; + for (let i = 0; i < str.length; i++) { + const c = str.charAt(i); + o.lines = (c === '\n' ? 1 : 0); + o.attribs = (c === 'a' || c === 'b' ? `*${c}` : ''); + assem.append(o); + } + return assem.toString(); + }; + + it(`testSplitJoinAttributionLines#${randomSeed}`, async function () { + const doc = `${randomMultiline(10, 20)}\n`; + + const theJoined = stringToOps(doc); + const theSplit = doc.match(/[^\n]*\n/g).map(stringToOps); + + expect(Changeset.splitAttributionLines(theJoined, doc)).to.eql(theSplit); + expect(Changeset.joinAttributionLines(theSplit)).to.equal(theJoined); + }); + }; + + for (let i = 0; i < 10; i++) testSplitJoinAttributionLines(i); + }); + }); +}); diff --git a/src/tests/frontend/specs/easysync-subAttribution.js b/src/tests/frontend/specs/easysync-subAttribution.js new file mode 100644 index 000000000..0cb4e8f7c --- /dev/null +++ b/src/tests/frontend/specs/easysync-subAttribution.js @@ -0,0 +1,55 @@ +'use strict'; + +const Changeset = require('../../../static/js/Changeset'); + +describe('easysync-subAttribution', function () { + const testSubattribution = (testId, astr, start, end, correctOutput) => { + it(`subattribution#${testId}`, async function () { + const str = Changeset.subattribution(astr, start, end); + expect(str).to.equal(correctOutput); + }); + }; + + testSubattribution(1, '+1', 0, 0, ''); + testSubattribution(2, '+1', 0, 1, '+1'); + testSubattribution(3, '+1', 0, undefined, '+1'); + testSubattribution(4, '|1+1', 0, 0, ''); + testSubattribution(5, '|1+1', 0, 1, '|1+1'); + testSubattribution(6, '|1+1', 0, undefined, '|1+1'); + testSubattribution(7, '*0+1', 0, 0, ''); + testSubattribution(8, '*0+1', 0, 1, '*0+1'); + testSubattribution(9, '*0+1', 0, undefined, '*0+1'); + testSubattribution(10, '*0|1+1', 0, 0, ''); + testSubattribution(11, '*0|1+1', 0, 1, '*0|1+1'); + testSubattribution(12, '*0|1+1', 0, undefined, '*0|1+1'); + testSubattribution(13, '*0+2+1*1+3', 0, 1, '*0+1'); + testSubattribution(14, '*0+2+1*1+3', 0, 2, '*0+2'); + testSubattribution(15, '*0+2+1*1+3', 0, 3, '*0+2+1'); + testSubattribution(16, '*0+2+1*1+3', 0, 4, '*0+2+1*1+1'); + testSubattribution(17, '*0+2+1*1+3', 0, 5, '*0+2+1*1+2'); + testSubattribution(18, '*0+2+1*1+3', 0, 6, '*0+2+1*1+3'); + testSubattribution(19, '*0+2+1*1+3', 0, 7, '*0+2+1*1+3'); + testSubattribution(20, '*0+2+1*1+3', 0, undefined, '*0+2+1*1+3'); + testSubattribution(21, '*0+2+1*1+3', 1, undefined, '*0+1+1*1+3'); + testSubattribution(22, '*0+2+1*1+3', 2, undefined, '+1*1+3'); + testSubattribution(23, '*0+2+1*1+3', 3, undefined, '*1+3'); + testSubattribution(24, '*0+2+1*1+3', 4, undefined, '*1+2'); + testSubattribution(25, '*0+2+1*1+3', 5, undefined, '*1+1'); + testSubattribution(26, '*0+2+1*1+3', 6, undefined, ''); + testSubattribution(27, '*0+2+1*1|1+3', 0, 1, '*0+1'); + testSubattribution(28, '*0+2+1*1|1+3', 0, 2, '*0+2'); + testSubattribution(29, '*0+2+1*1|1+3', 0, 3, '*0+2+1'); + testSubattribution(30, '*0+2+1*1|1+3', 0, 4, '*0+2+1*1+1'); + testSubattribution(31, '*0+2+1*1|1+3', 0, 5, '*0+2+1*1+2'); + testSubattribution(32, '*0+2+1*1|1+3', 0, 6, '*0+2+1*1|1+3'); + testSubattribution(33, '*0+2+1*1|1+3', 0, 7, '*0+2+1*1|1+3'); + testSubattribution(34, '*0+2+1*1|1+3', 0, undefined, '*0+2+1*1|1+3'); + testSubattribution(35, '*0+2+1*1|1+3', 1, undefined, '*0+1+1*1|1+3'); + testSubattribution(36, '*0+2+1*1|1+3', 2, undefined, '+1*1|1+3'); + testSubattribution(37, '*0+2+1*1|1+3', 3, undefined, '*1|1+3'); + testSubattribution(38, '*0+2+1*1|1+3', 4, undefined, '*1|1+2'); + testSubattribution(39, '*0+2+1*1|1+3', 5, undefined, '*1|1+1'); + testSubattribution(40, '*0+2+1*1|1+3', 1, 5, '*0+1+1*1+2'); + testSubattribution(41, '*0+2+1*1|1+3', 2, 6, '+1*1|1+3'); + testSubattribution(42, '*0+2+1*1+3', 2, 6, '+1*1+3'); +}); diff --git a/src/tests/frontend/specs/easysync.js b/src/tests/frontend/specs/easysync.js deleted file mode 100644 index 00ca58f45..000000000 --- a/src/tests/frontend/specs/easysync.js +++ /dev/null @@ -1,962 +0,0 @@ -'use strict'; -/** - * I found this tests in the old Etherpad and used it to test if the Changeset library can be run on - * node.js. It has no use for ep-lite, but I thought I keep it cause it may help someone to - * understand the Changeset library - * https://github.com/ether/pad/blob/master/infrastructure/ace/www/easysync2_tests.js - */ - -/* - * Copyright 2009 Google Inc., 2011 Peter 'Pita' Martischka (Primary Technology Ltd) - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS-IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - - -const Changeset = require('../../../static/js/Changeset'); -const AttributePool = require('../../../static/js/AttributePool'); - -const randInt = (maxValue) => Math.floor(Math.random() * maxValue); - -const poolOrArray = (attribs) => { - if (attribs.getAttrib) { - return attribs; // it's already an attrib pool - } else { - // assume it's an array of attrib strings to be split and added - const p = new AttributePool(); - attribs.forEach((kv) => { - p.putAttrib(kv.split(',')); - }); - return p; - } -}; - -const randomInlineString = (len) => { - const assem = Changeset.stringAssembler(); - for (let i = 0; i < len; i++) { - assem.append(String.fromCharCode(randInt(26) + 97)); - } - return assem.toString(); -}; - -const randomMultiline = (approxMaxLines, approxMaxCols) => { - const numParts = randInt(approxMaxLines * 2) + 1; - const txt = Changeset.stringAssembler(); - txt.append(randInt(2) ? '\n' : ''); - for (let i = 0; i < numParts; i++) { - if ((i % 2) === 0) { - if (randInt(10)) { - txt.append(randomInlineString(randInt(approxMaxCols) + 1)); - } else { - txt.append('\n'); - } - } else { - txt.append('\n'); - } - } - return txt.toString(); -}; - -const randomStringOperation = (numCharsLeft) => { - let result; - switch (randInt(9)) { - case 0: - { - // insert char - result = { - insert: randomInlineString(1), - }; - break; - } - case 1: - { - // delete char - result = { - remove: 1, - }; - break; - } - case 2: - { - // skip char - result = { - skip: 1, - }; - break; - } - case 3: - { - // insert small - result = { - insert: randomInlineString(randInt(4) + 1), - }; - break; - } - case 4: - { - // delete small - result = { - remove: randInt(4) + 1, - }; - break; - } - case 5: - { - // skip small - result = { - skip: randInt(4) + 1, - }; - break; - } - case 6: - { - // insert multiline; - result = { - insert: randomMultiline(5, 20), - }; - break; - } - case 7: - { - // delete multiline - result = { - remove: Math.round(numCharsLeft * Math.random() * Math.random()), - }; - break; - } - case 8: - { - // skip multiline - result = { - skip: Math.round(numCharsLeft * Math.random() * Math.random()), - }; - break; - } - case 9: - { - // delete to end - result = { - remove: numCharsLeft, - }; - break; - } - case 10: - { - // skip to end - result = { - skip: numCharsLeft, - }; - break; - } - } - const maxOrig = numCharsLeft - 1; - if ('remove' in result) { - result.remove = Math.min(result.remove, maxOrig); - } else if ('skip' in result) { - result.skip = Math.min(result.skip, maxOrig); - } - return result; -}; - -const randomTwoPropAttribs = (opcode) => { - // assumes attrib pool like ['apple,','apple,true','banana,','banana,true'] - if (opcode === '-' || randInt(3)) { - return ''; - } else if (randInt(3)) { // eslint-disable-line no-dupe-else-if - if (opcode === '+' || randInt(2)) { - return `*${Changeset.numToString(randInt(2) * 2 + 1)}`; - } else { - return `*${Changeset.numToString(randInt(2) * 2)}`; - } - } else if (opcode === '+' || randInt(4) === 0) { - return '*1*3'; - } else { - return ['*0*2', '*0*3', '*1*2'][randInt(3)]; - } -}; - -const randomTestChangeset = (origText, withAttribs) => { - const charBank = Changeset.stringAssembler(); - let textLeft = origText; // always keep final newline - const outTextAssem = Changeset.stringAssembler(); - const opAssem = Changeset.smartOpAssembler(); - const oldLen = origText.length; - - const nextOp = new Changeset.Op(); - - const appendMultilineOp = (opcode, txt) => { - nextOp.opcode = opcode; - if (withAttribs) { - nextOp.attribs = randomTwoPropAttribs(opcode); - } - txt.replace(/\n|[^\n]+/g, (t) => { - if (t === '\n') { - nextOp.chars = 1; - nextOp.lines = 1; - opAssem.append(nextOp); - } else { - nextOp.chars = t.length; - nextOp.lines = 0; - opAssem.append(nextOp); - } - return ''; - }); - }; - - const doOp = () => { - const o = randomStringOperation(textLeft.length); - if (o.insert) { - const txt = o.insert; - charBank.append(txt); - outTextAssem.append(txt); - appendMultilineOp('+', txt); - } else if (o.skip) { - const txt = textLeft.substring(0, o.skip); - textLeft = textLeft.substring(o.skip); - outTextAssem.append(txt); - appendMultilineOp('=', txt); - } else if (o.remove) { - const txt = textLeft.substring(0, o.remove); - textLeft = textLeft.substring(o.remove); - appendMultilineOp('-', txt); - } - }; - - while (textLeft.length > 1) doOp(); - for (let i = 0; i < 5; i++) doOp(); // do some more (only insertions will happen) - const outText = `${outTextAssem.toString()}\n`; - opAssem.endDocument(); - const cs = Changeset.pack(oldLen, outText.length, opAssem.toString(), charBank.toString()); - Changeset.checkRep(cs); - return [cs, outText]; -}; - -describe('easysync-assembler', function () { - it('opAssembler', async function () { - const x = '-c*3*4+6|3=az*asdf0*1*2*3+1=1-1+1*0+1=1-1+1|c=c-1'; - const assem = Changeset.opAssembler(); - for (const op of Changeset.deserializeOps(x)) assem.append(op); - expect(assem.toString()).to.equal(x); - }); - - it('smartOpAssembler', async function () { - const x = '-c*3*4+6|3=az*asdf0*1*2*3+1=1-1+1*0+1=1-1+1|c=c-1'; - const assem = Changeset.smartOpAssembler(); - for (const op of Changeset.deserializeOps(x)) assem.append(op); - assem.endDocument(); - expect(assem.toString()).to.equal(x); - }); - - describe('append atext to assembler', function () { - const testAppendATextToAssembler = (testId, atext, correctOps) => { - it(`testAppendATextToAssembler#${testId}`, async function () { - const assem = Changeset.smartOpAssembler(); - for (const op of Changeset.opsFromAText(atext)) assem.append(op); - expect(assem.toString()).to.equal(correctOps); - }); - }; - - testAppendATextToAssembler(1, { - text: '\n', - attribs: '|1+1', - }, ''); - testAppendATextToAssembler(2, { - text: '\n\n', - attribs: '|2+2', - }, '|1+1'); - testAppendATextToAssembler(3, { - text: '\n\n', - attribs: '*x|2+2', - }, '*x|1+1'); - testAppendATextToAssembler(4, { - text: '\n\n', - attribs: '*x|1+1|1+1', - }, '*x|1+1'); - testAppendATextToAssembler(5, { - text: 'foo\n', - attribs: '|1+4', - }, '+3'); - testAppendATextToAssembler(6, { - text: '\nfoo\n', - attribs: '|2+5', - }, '|1+1+3'); - testAppendATextToAssembler(7, { - text: '\nfoo\n', - attribs: '*x|2+5', - }, '*x|1+1*x+3'); - testAppendATextToAssembler(8, { - text: '\n\n\nfoo\n', - attribs: '|2+2*x|2+5', - }, '|2+2*x|1+1*x+3'); - }); -}); - -describe('easysync-compose', function () { - describe('compose', function () { - const testCompose = (randomSeed) => { - it(`testCompose#${randomSeed}`, async function () { - const p = new AttributePool(); - - const startText = `${randomMultiline(10, 20)}\n`; - - const x1 = randomTestChangeset(startText); - const change1 = x1[0]; - const text1 = x1[1]; - - const x2 = randomTestChangeset(text1); - const change2 = x2[0]; - const text2 = x2[1]; - - const x3 = randomTestChangeset(text2); - const change3 = x3[0]; - const text3 = x3[1]; - - const change12 = Changeset.checkRep(Changeset.compose(change1, change2, p)); - const change23 = Changeset.checkRep(Changeset.compose(change2, change3, p)); - const change123 = Changeset.checkRep(Changeset.compose(change12, change3, p)); - const change123a = Changeset.checkRep(Changeset.compose(change1, change23, p)); - expect(change123a).to.equal(change123); - - expect(Changeset.applyToText(change12, startText)).to.equal(text2); - expect(Changeset.applyToText(change23, text1)).to.equal(text3); - expect(Changeset.applyToText(change123, startText)).to.equal(text3); - }); - }; - - for (let i = 0; i < 30; i++) testCompose(i); - }); - - describe('compose attributes', function () { - it('simpleComposeAttributesTest', async function () { - const p = new AttributePool(); - p.putAttrib(['bold', '']); - p.putAttrib(['bold', 'true']); - const cs1 = Changeset.checkRep('Z:2>1*1+1*1=1$x'); - const cs2 = Changeset.checkRep('Z:3>0*0|1=3$'); - const cs12 = Changeset.checkRep(Changeset.compose(cs1, cs2, p)); - expect(cs12).to.equal('Z:2>1+1*0|1=2$x'); - }); - }); -}); - -describe('easysync-follow', function () { - describe('follow & compose', function () { - const testFollow = (randomSeed) => { - it(`testFollow#${randomSeed}`, async function () { - const p = new AttributePool(); - - const startText = `${randomMultiline(10, 20)}\n`; - - const cs1 = randomTestChangeset(startText)[0]; - const cs2 = randomTestChangeset(startText)[0]; - - const afb = Changeset.checkRep(Changeset.follow(cs1, cs2, false, p)); - const bfa = Changeset.checkRep(Changeset.follow(cs2, cs1, true, p)); - - const merge1 = Changeset.checkRep(Changeset.compose(cs1, afb)); - const merge2 = Changeset.checkRep(Changeset.compose(cs2, bfa)); - - expect(merge2).to.equal(merge1); - }); - }; - - for (let i = 0; i < 30; i++) testFollow(i); - }); - - describe('followAttributes & composeAttributes', function () { - const p = new AttributePool(); - p.putAttrib(['x', '']); - p.putAttrib(['x', 'abc']); - p.putAttrib(['x', 'def']); - p.putAttrib(['y', '']); - p.putAttrib(['y', 'abc']); - p.putAttrib(['y', 'def']); - let n = 0; - - const testFollow = (a, b, afb, bfa, merge) => { - it(`manual #${++n}`, async function () { - expect(Changeset.exportedForTestingOnly.followAttributes(a, b, p)).to.equal(afb); - expect(Changeset.exportedForTestingOnly.followAttributes(b, a, p)).to.equal(bfa); - expect(Changeset.composeAttributes(a, afb, true, p)).to.equal(merge); - expect(Changeset.composeAttributes(b, bfa, true, p)).to.equal(merge); - }); - }; - - testFollow('', '', '', '', ''); - testFollow('*0', '', '', '*0', '*0'); - testFollow('*0', '*0', '', '', '*0'); - testFollow('*0', '*1', '', '*0', '*0'); - testFollow('*1', '*2', '', '*1', '*1'); - testFollow('*0*1', '', '', '*0*1', '*0*1'); - testFollow('*0*4', '*2*3', '*3', '*0', '*0*3'); - testFollow('*0*4', '*2', '', '*0*4', '*0*4'); - }); - - describe('chracterRangeFollow', function () { - const testCharacterRangeFollow = (testId, cs, oldRange, insertionsAfter, correctNewRange) => { - it(`testCharacterRangeFollow#${testId}`, async function () { - cs = Changeset.checkRep(cs); - expect(Changeset.characterRangeFollow(cs, oldRange[0], oldRange[1], insertionsAfter)) - .to.eql(correctNewRange); - }); - }; - - testCharacterRangeFollow(1, 'Z:z>9*0=1=4-3+9=1|1-4-4+1*0+a$123456789abcdefghijk', - [7, 10], false, [14, 15]); - testCharacterRangeFollow(2, 'Z:bc<6|x=b4|2-6$', [400, 407], false, [400, 401]); - testCharacterRangeFollow(3, 'Z:4>0-3+3$abc', [0, 3], false, [3, 3]); - testCharacterRangeFollow(4, 'Z:4>0-3+3$abc', [0, 3], true, [0, 0]); - testCharacterRangeFollow(5, 'Z:5>1+1=1-3+3$abcd', [1, 4], false, [5, 5]); - testCharacterRangeFollow(6, 'Z:5>1+1=1-3+3$abcd', [1, 4], true, [2, 2]); - testCharacterRangeFollow(7, 'Z:5>1+1=1-3+3$abcd', [0, 6], false, [1, 7]); - testCharacterRangeFollow(8, 'Z:5>1+1=1-3+3$abcd', [0, 3], false, [1, 2]); - testCharacterRangeFollow(9, 'Z:5>1+1=1-3+3$abcd', [2, 5], false, [5, 6]); - testCharacterRangeFollow(10, 'Z:2>1+1$a', [0, 0], false, [1, 1]); - testCharacterRangeFollow(11, 'Z:2>1+1$a', [0, 0], true, [0, 0]); - }); -}); - -describe('easysync-inverseRandom', function () { - describe('inverse random', function () { - const testInverseRandom = (randomSeed) => { - it(`testInverseRandom#${randomSeed}`, async function () { - const p = poolOrArray(['apple,', 'apple,true', 'banana,', 'banana,true']); - - const startText = `${randomMultiline(10, 20)}\n`; - const alines = - Changeset.splitAttributionLines(Changeset.makeAttribution(startText), startText); - const lines = startText.slice(0, -1).split('\n').map((s) => `${s}\n`); - - const stylifier = randomTestChangeset(startText, true)[0]; - - Changeset.mutateAttributionLines(stylifier, alines, p); - Changeset.mutateTextLines(stylifier, lines); - - const changeset = randomTestChangeset(lines.join(''), true)[0]; - const inverseChangeset = Changeset.inverse(changeset, lines, alines, p); - - const origLines = lines.slice(); - const origALines = alines.slice(); - - Changeset.mutateTextLines(changeset, lines); - Changeset.mutateAttributionLines(changeset, alines, p); - Changeset.mutateTextLines(inverseChangeset, lines); - Changeset.mutateAttributionLines(inverseChangeset, alines, p); - expect(lines).to.eql(origLines); - expect(alines).to.eql(origALines); - }); - }; - - for (let i = 0; i < 30; i++) testInverseRandom(i); - }); - - describe('inverse', function () { - const testInverse = (testId, cs, lines, alines, pool, correctOutput) => { - it(`testInverse#${testId}`, async function () { - pool = poolOrArray(pool); - const str = Changeset.inverse(Changeset.checkRep(cs), lines, alines, pool); - expect(str).to.equal(correctOutput); - }); - }; - - // take "FFFFTTTTT" and apply "-FT--FFTT", the inverse of which is "--F--TT--" - testInverse(1, 'Z:9>0=1*0=1*1=1=2*0=2*1|1=2$', null, - ['+4*1+5'], ['bold,', 'bold,true'], 'Z:9>0=2*0=1=2*1=2$'); - }); -}); - -describe('easysync-mutations', function () { - const applyMutations = (mu, arrayOfArrays) => { - arrayOfArrays.forEach((a) => { - const result = mu[a[0]](...a.slice(1)); - if (a[0] === 'remove' && a[3]) { - expect(result).to.equal(a[3]); - } - }); - }; - - const mutationsToChangeset = (oldLen, arrayOfArrays) => { - const assem = Changeset.smartOpAssembler(); - const op = new Changeset.Op(); - const bank = Changeset.stringAssembler(); - let oldPos = 0; - let newLen = 0; - arrayOfArrays.forEach((a) => { - if (a[0] === 'skip') { - op.opcode = '='; - op.chars = a[1]; - op.lines = (a[2] || 0); - assem.append(op); - oldPos += op.chars; - newLen += op.chars; - } else if (a[0] === 'remove') { - op.opcode = '-'; - op.chars = a[1]; - op.lines = (a[2] || 0); - assem.append(op); - oldPos += op.chars; - } else if (a[0] === 'insert') { - op.opcode = '+'; - bank.append(a[1]); - op.chars = a[1].length; - op.lines = (a[2] || 0); - assem.append(op); - newLen += op.chars; - } - }); - newLen += oldLen - oldPos; - assem.endDocument(); - return Changeset.pack(oldLen, newLen, assem.toString(), bank.toString()); - }; - - const runMutationTest = (testId, origLines, muts, correct) => { - it(`runMutationTest#${testId}`, async function () { - let lines = origLines.slice(); - const mu = new Changeset.exportedForTestingOnly.TextLinesMutator(lines); - applyMutations(mu, muts); - mu.close(); - expect(lines).to.eql(correct); - - const inText = origLines.join(''); - const cs = mutationsToChangeset(inText.length, muts); - lines = origLines.slice(); - Changeset.mutateTextLines(cs, lines); - expect(lines).to.eql(correct); - - const correctText = correct.join(''); - const outText = Changeset.applyToText(cs, inText); - expect(outText).to.equal(correctText); - }); - }; - - runMutationTest(1, ['apple\n', 'banana\n', 'cabbage\n', 'duffle\n', 'eggplant\n'], [ - ['remove', 1, 0, 'a'], - ['insert', 'tu'], - ['remove', 1, 0, 'p'], - ['skip', 4, 1], - ['skip', 7, 1], - ['insert', 'cream\npie\n', 2], - ['skip', 2], - ['insert', 'bot'], - ['insert', '\n', 1], - ['insert', 'bu'], - ['skip', 3], - ['remove', 3, 1, 'ge\n'], - ['remove', 6, 0, 'duffle'], - ], ['tuple\n', 'banana\n', 'cream\n', 'pie\n', 'cabot\n', 'bubba\n', 'eggplant\n']); - - runMutationTest(2, ['apple\n', 'banana\n', 'cabbage\n', 'duffle\n', 'eggplant\n'], [ - ['remove', 1, 0, 'a'], - ['remove', 1, 0, 'p'], - ['insert', 'tu'], - ['skip', 11, 2], - ['insert', 'cream\npie\n', 2], - ['skip', 2], - ['insert', 'bot'], - ['insert', '\n', 1], - ['insert', 'bu'], - ['skip', 3], - ['remove', 3, 1, 'ge\n'], - ['remove', 6, 0, 'duffle'], - ], ['tuple\n', 'banana\n', 'cream\n', 'pie\n', 'cabot\n', 'bubba\n', 'eggplant\n']); - - runMutationTest(3, ['apple\n', 'banana\n', 'cabbage\n', 'duffle\n', 'eggplant\n'], [ - ['remove', 6, 1, 'apple\n'], - ['skip', 15, 2], - ['skip', 6], - ['remove', 1, 1, '\n'], - ['remove', 8, 0, 'eggplant'], - ['skip', 1, 1], - ], ['banana\n', 'cabbage\n', 'duffle\n']); - - runMutationTest(4, ['15\n'], [ - ['skip', 1], - ['insert', '\n2\n3\n4\n', 4], - ['skip', 2, 1], - ], ['1\n', '2\n', '3\n', '4\n', '5\n']); - - runMutationTest(5, ['1\n', '2\n', '3\n', '4\n', '5\n'], [ - ['skip', 1], - ['remove', 7, 4, '\n2\n3\n4\n'], - ['skip', 2, 1], - ], ['15\n']); - - runMutationTest(6, ['123\n', 'abc\n', 'def\n', 'ghi\n', 'xyz\n'], [ - ['insert', '0'], - ['skip', 4, 1], - ['skip', 4, 1], - ['remove', 8, 2, 'def\nghi\n'], - ['skip', 4, 1], - ], ['0123\n', 'abc\n', 'xyz\n']); - - runMutationTest(7, ['apple\n', 'banana\n', 'cabbage\n', 'duffle\n', 'eggplant\n'], [ - ['remove', 6, 1, 'apple\n'], - ['skip', 15, 2, true], - ['skip', 6, 0, true], - ['remove', 1, 1, '\n'], - ['remove', 8, 0, 'eggplant'], - ['skip', 1, 1, true], - ], ['banana\n', 'cabbage\n', 'duffle\n']); - - it('mutatorHasMore', async function () { - const lines = ['1\n', '2\n', '3\n', '4\n']; - let mu; - - mu = new Changeset.exportedForTestingOnly.TextLinesMutator(lines); - expect(mu.hasMore()).to.be(true); - mu.skip(8, 4); - expect(mu.hasMore()).to.be(false); - mu.close(); - expect(mu.hasMore()).to.be(false); - - // still 1,2,3,4 - mu = new Changeset.exportedForTestingOnly.TextLinesMutator(lines); - expect(mu.hasMore()).to.be(true); - mu.remove(2, 1); - expect(mu.hasMore()).to.be(true); - mu.skip(2, 1); - expect(mu.hasMore()).to.be(true); - mu.skip(2, 1); - expect(mu.hasMore()).to.be(true); - mu.skip(2, 1); - expect(mu.hasMore()).to.be(false); - mu.insert('5\n', 1); - expect(mu.hasMore()).to.be(false); - mu.close(); - expect(mu.hasMore()).to.be(false); - - // 2,3,4,5 now - mu = new Changeset.exportedForTestingOnly.TextLinesMutator(lines); - expect(mu.hasMore()).to.be(true); - mu.remove(6, 3); - expect(mu.hasMore()).to.be(true); - mu.remove(2, 1); - expect(mu.hasMore()).to.be(false); - mu.insert('hello\n', 1); - expect(mu.hasMore()).to.be(false); - mu.close(); - expect(mu.hasMore()).to.be(false); - }); - - describe('mutateTextLines', function () { - const testMutateTextLines = (testId, cs, lines, correctLines) => { - it(`testMutateTextLines#${testId}`, async function () { - const a = lines.slice(); - Changeset.mutateTextLines(cs, a); - expect(a).to.eql(correctLines); - }); - }; - - testMutateTextLines(1, 'Z:4<1|1-2-1|1+1+1$\nc', ['a\n', 'b\n'], ['\n', 'c\n']); - testMutateTextLines(2, 'Z:4>0|1-2-1|2+3$\nc\n', ['a\n', 'b\n'], ['\n', 'c\n', '\n']); - }); - - describe('mutate attributions', function () { - const testPoolWithChars = (() => { - const p = new AttributePool(); - p.putAttrib(['char', 'newline']); - for (let i = 1; i < 36; i++) { - p.putAttrib(['char', Changeset.numToString(i)]); - } - p.putAttrib(['char', '']); - return p; - })(); - - const runMutateAttributionTest = (testId, attribs, cs, alines, outCorrect) => { - it(`runMutateAttributionTest#${testId}`, async function () { - const p = poolOrArray(attribs); - const alines2 = Array.prototype.slice.call(alines); - Changeset.mutateAttributionLines(Changeset.checkRep(cs), alines2, p); - expect(alines2).to.eql(outCorrect); - - const removeQuestionMarks = (a) => a.replace(/\?/g, ''); - const inMerged = Changeset.joinAttributionLines(alines.map(removeQuestionMarks)); - const correctMerged = Changeset.joinAttributionLines(outCorrect.map(removeQuestionMarks)); - const mergedResult = Changeset.applyToAttribution(cs, inMerged, p); - expect(mergedResult).to.equal(correctMerged); - }); - }; - - // turn 123\n 456\n 789\n into 123\n 456\n 789\n - runMutateAttributionTest(1, - ['bold,true'], 'Z:c>0|1=4=1*0=1$', ['|1+4', '|1+4', '|1+4'], ['|1+4', '+1*0+1|1+2', '|1+4'] - ); - - // make a document bold - runMutateAttributionTest(2, - ['bold,true'], 'Z:c>0*0|3=c$', ['|1+4', '|1+4', '|1+4'], ['*0|1+4', '*0|1+4', '*0|1+4']); - - // clear bold on document - runMutateAttributionTest(3, - ['bold,', 'bold,true'], 'Z:c>0*0|3=c$', - ['*1+1+1*1+1|1+1', '+1*1+1|1+2', '*1+1+1*1+1|1+1'], ['|1+4', '|1+4', '|1+4']); - - // add a character on line 3 of a document with 5 blank lines, and make sure - // the optimization that skips purely-kept lines is working; if any attribution string - // with a '?' is parsed it will cause an error. - runMutateAttributionTest(4, - ['foo,bar', 'line,1', 'line,2', 'line,3', 'line,4', 'line,5'], - 'Z:5>1|2=2+1$x', ['?*1|1+1', '?*2|1+1', '*3|1+1', '?*4|1+1', '?*5|1+1'], - ['?*1|1+1', '?*2|1+1', '+1*3|1+1', '?*4|1+1', '?*5|1+1']); - - // based on runMutationTest#1 - runMutateAttributionTest(5, testPoolWithChars, - 'Z:11>7-2*t+1*u+1|2=b|2+a=2*b+1*o+1*t+1*0|1+1*b+1*u+1=3|1-3-6$tucream\npie\nbot\nbu', - [ - '*a+1*p+2*l+1*e+1*0|1+1', - '*b+1*a+1*n+1*a+1*n+1*a+1*0|1+1', - '*c+1*a+1*b+2*a+1*g+1*e+1*0|1+1', - '*d+1*u+1*f+2*l+1*e+1*0|1+1', - '*e+1*g+2*p+1*l+1*a+1*n+1*t+1*0|1+1', - ], - [ - '*t+1*u+1*p+1*l+1*e+1*0|1+1', - '*b+1*a+1*n+1*a+1*n+1*a+1*0|1+1', - '|1+6', - '|1+4', - '*c+1*a+1*b+1*o+1*t+1*0|1+1', - '*b+1*u+1*b+2*a+1*0|1+1', - '*e+1*g+2*p+1*l+1*a+1*n+1*t+1*0|1+1', - ]); - - // based on runMutationTest#3 - runMutateAttributionTest(6, testPoolWithChars, - 'Z:117=1|4+7$\n2\n3\n4\n', - ['*1+1*5|1+2'], ['*1+1|1+1', '|1+2', '|1+2', '|1+2', '*5|1+2']); - - // based on runMutationTest#5 - runMutateAttributionTest(8, testPoolWithChars, 'Z:a<7=1|4-7$', - ['*1|1+2', '*2|1+2', '*3|1+2', '*4|1+2', '*5|1+2'], ['*1+1*5|1+2']); - - // based on runMutationTest#6 - runMutateAttributionTest(9, testPoolWithChars, 'Z:k<7*0+1*10|2=8|2-8$0', - [ - '*1+1*2+1*3+1|1+1', - '*a+1*b+1*c+1|1+1', - '*d+1*e+1*f+1|1+1', - '*g+1*h+1*i+1|1+1', - '?*x+1*y+1*z+1|1+1', - ], - ['*0+1|1+4', '|1+4', '?*x+1*y+1*z+1|1+1']); - - runMutateAttributionTest(10, testPoolWithChars, 'Z:6>4=1+1=1+1|1=1+1=1*0+1$abcd', - ['|1+3', '|1+3'], ['|1+5', '+2*0+1|1+2']); - - runMutateAttributionTest(11, testPoolWithChars, 'Z:s>1|1=4=6|1+1$\n', - ['*0|1+4', '*0|1+8', '*0+5|1+1', '*0|1+1', '*0|1+5', '*0|1+1', '*0|1+1', '*0|1+1', '|1+1'], - [ - '*0|1+4', - '*0+6|1+1', - '*0|1+2', - '*0+5|1+1', - '*0|1+1', - '*0|1+5', - '*0|1+1', - '*0|1+1', - '*0|1+1', - '|1+1', - ]); - }); -}); - -describe('easysync-other', function () { - describe('filter attribute numbers', function () { - const testFilterAttribNumbers = (testId, cs, filter, correctOutput) => { - it(`testFilterAttribNumbers#${testId}`, async function () { - const str = Changeset.filterAttribNumbers(cs, filter); - expect(str).to.equal(correctOutput); - }); - }; - - testFilterAttribNumbers(1, '*0*1+1+2+3*1+4*2+5*0*2*1*b*c+6', - (n) => (n % 2) === 0, '*0+1+2+3+4*2+5*0*2*c+6'); - testFilterAttribNumbers(2, '*0*1+1+2+3*1+4*2+5*0*2*1*b*c+6', - (n) => (n % 2) === 1, '*1+1+2+3*1+4+5*1*b+6'); - }); - - describe('make attribs string', function () { - const testMakeAttribsString = (testId, pool, opcode, attribs, correctString) => { - it(`testMakeAttribsString#${testId}`, async function () { - const p = poolOrArray(pool); - const str = Changeset.makeAttribsString(opcode, attribs, p); - expect(str).to.equal(correctString); - }); - }; - - testMakeAttribsString(1, ['bold,'], '+', [ - ['bold', ''], - ], ''); - testMakeAttribsString(2, ['abc,def', 'bold,'], '=', [ - ['bold', ''], - ], '*1'); - testMakeAttribsString(3, ['abc,def', 'bold,true'], '+', [ - ['abc', 'def'], - ['bold', 'true'], - ], '*0*1'); - testMakeAttribsString(4, ['abc,def', 'bold,true'], '+', [ - ['bold', 'true'], - ['abc', 'def'], - ], '*0*1'); - }); - - describe('other', function () { - it('testMoveOpsToNewPool', async function () { - const pool1 = new AttributePool(); - const pool2 = new AttributePool(); - - pool1.putAttrib(['baz', 'qux']); - pool1.putAttrib(['foo', 'bar']); - - pool2.putAttrib(['foo', 'bar']); - - expect(Changeset.moveOpsToNewPool('Z:1>2*1+1*0+1$ab', pool1, pool2)) - .to.equal('Z:1>2*0+1*1+1$ab'); - expect(Changeset.moveOpsToNewPool('*1+1*0+1', pool1, pool2)).to.equal('*0+1*1+1'); - }); - - it('testMakeSplice', async function () { - const t = 'a\nb\nc\n'; - const t2 = Changeset.applyToText(Changeset.makeSplice(t, 5, 0, 'def'), t); - expect(t2).to.equal('a\nb\ncdef\n'); - }); - - it('testToSplices', async function () { - const cs = Changeset.checkRep('Z:z>9*0=1=4-3+9=1|1-4-4+1*0+a$123456789abcdefghijk'); - const correctSplices = [ - [5, 8, '123456789'], - [9, 17, 'abcdefghijk'], - ]; - expect(Changeset.exportedForTestingOnly.toSplices(cs)).to.eql(correctSplices); - }); - - it('opAttributeValue', async function () { - const p = new AttributePool(); - p.putAttrib(['name', 'david']); - p.putAttrib(['color', 'green']); - - const stringOp = (str) => Changeset.deserializeOps(str).next().value; - - expect(Changeset.opAttributeValue(stringOp('*0*1+1'), 'name', p)).to.equal('david'); - expect(Changeset.opAttributeValue(stringOp('*0+1'), 'name', p)).to.equal('david'); - expect(Changeset.opAttributeValue(stringOp('*1+1'), 'name', p)).to.equal(''); - expect(Changeset.opAttributeValue(stringOp('+1'), 'name', p)).to.equal(''); - expect(Changeset.opAttributeValue(stringOp('*0*1+1'), 'color', p)).to.equal('green'); - expect(Changeset.opAttributeValue(stringOp('*1+1'), 'color', p)).to.equal('green'); - expect(Changeset.opAttributeValue(stringOp('*0+1'), 'color', p)).to.equal(''); - expect(Changeset.opAttributeValue(stringOp('+1'), 'color', p)).to.equal(''); - }); - - describe('applyToAttribution', function () { - const runApplyToAttributionTest = (testId, attribs, cs, inAttr, outCorrect) => { - it(`applyToAttribution#${testId}`, async function () { - const p = poolOrArray(attribs); - const result = Changeset.applyToAttribution(Changeset.checkRep(cs), inAttr, p); - expect(result).to.equal(outCorrect); - }); - }; - - // turn cactus\n into actusabcd\n - runApplyToAttributionTest(1, - ['bold,', 'bold,true'], 'Z:7>3-1*0=1*1=1=3+4$abcd', '+1*1+1|1+5', '+1*1+1|1+8'); - - // turn "david\ngreenspan\n" into "david\ngreen\n" - runApplyToAttributionTest(2, - ['bold,', 'bold,true'], 'Z:g<4*1|1=6*1=5-4$', '|2+g', '*1|1+6*1+5|1+1'); - }); - - describe('split/join attribution lines', function () { - const testSplitJoinAttributionLines = (randomSeed) => { - const stringToOps = (str) => { - const assem = Changeset.mergingOpAssembler(); - const o = new Changeset.Op('+'); - o.chars = 1; - for (let i = 0; i < str.length; i++) { - const c = str.charAt(i); - o.lines = (c === '\n' ? 1 : 0); - o.attribs = (c === 'a' || c === 'b' ? `*${c}` : ''); - assem.append(o); - } - return assem.toString(); - }; - - it(`testSplitJoinAttributionLines#${randomSeed}`, async function () { - const doc = `${randomMultiline(10, 20)}\n`; - - const theJoined = stringToOps(doc); - const theSplit = doc.match(/[^\n]*\n/g).map(stringToOps); - - expect(Changeset.splitAttributionLines(theJoined, doc)).to.eql(theSplit); - expect(Changeset.joinAttributionLines(theSplit)).to.equal(theJoined); - }); - }; - - for (let i = 0; i < 10; i++) testSplitJoinAttributionLines(i); - }); - }); -}); - -describe('easysync-subAttribution', function () { - const testSubattribution = (testId, astr, start, end, correctOutput) => { - it(`subattribution#${testId}`, async function () { - const str = Changeset.subattribution(astr, start, end); - expect(str).to.equal(correctOutput); - }); - }; - - testSubattribution(1, '+1', 0, 0, ''); - testSubattribution(2, '+1', 0, 1, '+1'); - testSubattribution(3, '+1', 0, undefined, '+1'); - testSubattribution(4, '|1+1', 0, 0, ''); - testSubattribution(5, '|1+1', 0, 1, '|1+1'); - testSubattribution(6, '|1+1', 0, undefined, '|1+1'); - testSubattribution(7, '*0+1', 0, 0, ''); - testSubattribution(8, '*0+1', 0, 1, '*0+1'); - testSubattribution(9, '*0+1', 0, undefined, '*0+1'); - testSubattribution(10, '*0|1+1', 0, 0, ''); - testSubattribution(11, '*0|1+1', 0, 1, '*0|1+1'); - testSubattribution(12, '*0|1+1', 0, undefined, '*0|1+1'); - testSubattribution(13, '*0+2+1*1+3', 0, 1, '*0+1'); - testSubattribution(14, '*0+2+1*1+3', 0, 2, '*0+2'); - testSubattribution(15, '*0+2+1*1+3', 0, 3, '*0+2+1'); - testSubattribution(16, '*0+2+1*1+3', 0, 4, '*0+2+1*1+1'); - testSubattribution(17, '*0+2+1*1+3', 0, 5, '*0+2+1*1+2'); - testSubattribution(18, '*0+2+1*1+3', 0, 6, '*0+2+1*1+3'); - testSubattribution(19, '*0+2+1*1+3', 0, 7, '*0+2+1*1+3'); - testSubattribution(20, '*0+2+1*1+3', 0, undefined, '*0+2+1*1+3'); - testSubattribution(21, '*0+2+1*1+3', 1, undefined, '*0+1+1*1+3'); - testSubattribution(22, '*0+2+1*1+3', 2, undefined, '+1*1+3'); - testSubattribution(23, '*0+2+1*1+3', 3, undefined, '*1+3'); - testSubattribution(24, '*0+2+1*1+3', 4, undefined, '*1+2'); - testSubattribution(25, '*0+2+1*1+3', 5, undefined, '*1+1'); - testSubattribution(26, '*0+2+1*1+3', 6, undefined, ''); - testSubattribution(27, '*0+2+1*1|1+3', 0, 1, '*0+1'); - testSubattribution(28, '*0+2+1*1|1+3', 0, 2, '*0+2'); - testSubattribution(29, '*0+2+1*1|1+3', 0, 3, '*0+2+1'); - testSubattribution(30, '*0+2+1*1|1+3', 0, 4, '*0+2+1*1+1'); - testSubattribution(31, '*0+2+1*1|1+3', 0, 5, '*0+2+1*1+2'); - testSubattribution(32, '*0+2+1*1|1+3', 0, 6, '*0+2+1*1|1+3'); - testSubattribution(33, '*0+2+1*1|1+3', 0, 7, '*0+2+1*1|1+3'); - testSubattribution(34, '*0+2+1*1|1+3', 0, undefined, '*0+2+1*1|1+3'); - testSubattribution(35, '*0+2+1*1|1+3', 1, undefined, '*0+1+1*1|1+3'); - testSubattribution(36, '*0+2+1*1|1+3', 2, undefined, '+1*1|1+3'); - testSubattribution(37, '*0+2+1*1|1+3', 3, undefined, '*1|1+3'); - testSubattribution(38, '*0+2+1*1|1+3', 4, undefined, '*1|1+2'); - testSubattribution(39, '*0+2+1*1|1+3', 5, undefined, '*1|1+1'); - testSubattribution(40, '*0+2+1*1|1+3', 1, 5, '*0+1+1*1+2'); - testSubattribution(41, '*0+2+1*1|1+3', 2, 6, '+1*1|1+3'); - testSubattribution(42, '*0+2+1*1+3', 2, 6, '+1*1+3'); -}); From 09c9e32d72ddd85c29097484317234e1f4c2c995 Mon Sep 17 00:00:00 2001 From: Timon Engelke Date: Fri, 12 Nov 2021 23:16:20 +0100 Subject: [PATCH 035/446] Delete session after corresponding group2session and author2session --- src/node/db/SessionManager.js | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/src/node/db/SessionManager.js b/src/node/db/SessionManager.js index b5f93094d..c614aaf87 100644 --- a/src/node/db/SessionManager.js +++ b/src/node/db/SessionManager.js @@ -204,9 +204,6 @@ exports.deleteSession = async (sessionID) => { const group2sessions = await db.get(`group2sessions:${groupID}`); const author2sessions = await db.get(`author2sessions:${authorID}`); - // remove the session - await db.remove(`session:${sessionID}`); - // remove session from group2sessions if (group2sessions != null) { // Maybe the group was already deleted delete group2sessions.sessionIDs[sessionID]; @@ -218,6 +215,9 @@ exports.deleteSession = async (sessionID) => { delete author2sessions.sessionIDs[sessionID]; await db.set(`author2sessions:${authorID}`, author2sessions); } + + // remove the session + await db.remove(`session:${sessionID}`); }; exports.listSessionsOfGroup = async (groupID) => { From 3070cee9ca0b4735ecce2a9b74d5b915bf0a2439 Mon Sep 17 00:00:00 2001 From: Timon Engelke Date: Sat, 13 Nov 2021 00:15:31 +0100 Subject: [PATCH 036/446] Delete group after removing it from the group list --- src/node/db/GroupManager.js | 22 ++++++++++------------ 1 file changed, 10 insertions(+), 12 deletions(-) diff --git a/src/node/db/GroupManager.js b/src/node/db/GroupManager.js index 203e21a35..18f91f566 100644 --- a/src/node/db/GroupManager.js +++ b/src/node/db/GroupManager.js @@ -55,9 +55,8 @@ exports.deleteGroup = async (groupID) => { // loop through all sessions and delete them (in parallel) await Promise.all(Object.keys(sessions).map((session) => sessionManager.deleteSession(session))); - // remove group and group2sessions entry + // remove group2sessions entry await db.remove(`group2sessions:${groupID}`); - await db.remove(`group:${groupID}`); // unlist the group let groups = await exports.listAllGroups(); @@ -65,19 +64,18 @@ exports.deleteGroup = async (groupID) => { const index = groups.indexOf(groupID); - if (index === -1) { - // it's not listed + if (index !== -1) { + // remove from the list + groups.splice(index, 1); - return; + // regenerate group list + const newGroups = {}; + groups.forEach((group) => newGroups[group] = 1); + await db.set('groups', newGroups); } - // remove from the list - groups.splice(index, 1); - - // regenerate group list - const newGroups = {}; - groups.forEach((group) => newGroups[group] = 1); - await db.set('groups', newGroups); + // remove group entry + await db.remove(`group:${groupID}`); }; exports.doesGroupExist = async (groupID) => { From 9d63700da0d5e237e7d6089deb9f6f0cb14df974 Mon Sep 17 00:00:00 2001 From: Richard Hansen Date: Fri, 12 Nov 2021 21:26:01 -0500 Subject: [PATCH 037/446] SessionManager: Use `.setSub()` and parallel queries to avoid races This also simplfies the code. --- src/node/db/SessionManager.js | 68 +++++++++-------------------------- 1 file changed, 17 insertions(+), 51 deletions(-) diff --git a/src/node/db/SessionManager.js b/src/node/db/SessionManager.js index c614aaf87..5ef75acfa 100644 --- a/src/node/db/SessionManager.js +++ b/src/node/db/SessionManager.js @@ -134,41 +134,14 @@ exports.createSession = async (groupID, authorID, validUntil) => { // set the session into the database await db.set(`session:${sessionID}`, {groupID, authorID, validUntil}); - // get the entry - let group2sessions = await db.get(`group2sessions:${groupID}`); - - /* - * In some cases, the db layer could return "undefined" as well as "null". - * Thus, it is not possible to perform strict null checks on group2sessions. - * In a previous version of this code, a strict check broke session - * management. - * - * See: https://github.com/ether/etherpad-lite/issues/3567#issuecomment-468613960 - */ - if (!group2sessions || !group2sessions.sessionIDs) { - // the entry doesn't exist so far, let's create it - group2sessions = {sessionIDs: {}}; - } - - // add the entry for this session - group2sessions.sessionIDs[sessionID] = 1; - - // save the new element back - await db.set(`group2sessions:${groupID}`, group2sessions); - - // get the author2sessions entry - let author2sessions = await db.get(`author2sessions:${authorID}`); - - if (author2sessions == null || author2sessions.sessionIDs == null) { - // the entry doesn't exist so far, let's create it - author2sessions = {sessionIDs: {}}; - } - - // add the entry for this session - author2sessions.sessionIDs[sessionID] = 1; - - // save the new element back - await db.set(`author2sessions:${authorID}`, author2sessions); + // Add the session ID to the group2sessions and author2sessions records after creating the session + // so that the state is consistent. + await Promise.all([ + // UeberDB's setSub() method atomically reads the record, updates the appropriate (sub)object + // property, and writes the result. + db.setSub(`group2sessions:${groupID}`, ['sessionIDs', sessionID], 1), + db.setSub(`author2sessions:${authorID}`, ['sessionIDs', sessionID], 1), + ]); return {sessionID}; }; @@ -200,23 +173,16 @@ exports.deleteSession = async (sessionID) => { const groupID = session.groupID; const authorID = session.authorID; - // get the group2sessions and author2sessions entries - const group2sessions = await db.get(`group2sessions:${groupID}`); - const author2sessions = await db.get(`author2sessions:${authorID}`); + await Promise.all([ + // UeberDB's setSub() method atomically reads the record, updates the appropriate (sub)object + // property, and writes the result. Setting a property to `undefined` deletes that property + // (JSON.stringify() ignores such properties). + db.setSub(`group2sessions:${groupID}`, ['sessionIDs', sessionID], undefined), + db.setSub(`author2sessions:${authorID}`, ['sessionIDs', sessionID], undefined), + ]); - // remove session from group2sessions - if (group2sessions != null) { // Maybe the group was already deleted - delete group2sessions.sessionIDs[sessionID]; - await db.set(`group2sessions:${groupID}`, group2sessions); - } - - // remove session from author2sessions - if (author2sessions != null) { // Maybe the author was already deleted - delete author2sessions.sessionIDs[sessionID]; - await db.set(`author2sessions:${authorID}`, author2sessions); - } - - // remove the session + // Delete the session record after updating group2sessions and author2sessions so that the state + // is consistent. await db.remove(`session:${sessionID}`); }; From 5b37a5619781012decb3f5aae847de447aa5f08d Mon Sep 17 00:00:00 2001 From: Richard Hansen Date: Fri, 12 Nov 2021 21:26:55 -0500 Subject: [PATCH 038/446] GroupManager: Use `.setSub()` and parallel queries to avoid races This also simplfies the code. --- src/node/db/GroupManager.js | 65 +++++++++++++------------------------ 1 file changed, 22 insertions(+), 43 deletions(-) diff --git a/src/node/db/GroupManager.js b/src/node/db/GroupManager.js index 18f91f566..de5efc2e4 100644 --- a/src/node/db/GroupManager.js +++ b/src/node/db/GroupManager.js @@ -43,38 +43,27 @@ exports.deleteGroup = async (groupID) => { } // iterate through all pads of this group and delete them (in parallel) - await Promise.all(Object.keys(group.pads) - .map((padID) => padManager.getPad(padID) - .then((pad) => pad.remove()) - )); + await Promise.all(Object.keys(group.pads).map(async (padId) => { + const pad = await padManager.getPad(padId); + await pad.remove(); + })); - // iterate through group2sessions and delete all sessions - const group2sessions = await db.get(`group2sessions:${groupID}`); - const sessions = group2sessions ? group2sessions.sessionIDs : {}; + // Delete associated sessions in parallel. This should be done before deleting the group2sessions + // record because deleting a session updates the group2sessions record. + const {sessionIDs = {}} = await db.get(`group2sessions:${groupID}`) || {}; + await Promise.all(Object.keys(sessionIDs).map(async (sessionId) => { + await sessionManager.deleteSession(sessionId); + })); - // loop through all sessions and delete them (in parallel) - await Promise.all(Object.keys(sessions).map((session) => sessionManager.deleteSession(session))); + await Promise.all([ + db.remove(`group2sessions:${groupID}`), + // UeberDB's setSub() method atomically reads the record, updates the appropriate property, and + // writes the result. Setting a property to `undefined` deletes that property (JSON.stringify() + // ignores such properties). + db.setSub('groups', [groupID], undefined), + ]); - // remove group2sessions entry - await db.remove(`group2sessions:${groupID}`); - - // unlist the group - let groups = await exports.listAllGroups(); - groups = groups ? groups.groupIDs : []; - - const index = groups.indexOf(groupID); - - if (index !== -1) { - // remove from the list - groups.splice(index, 1); - - // regenerate group list - const newGroups = {}; - groups.forEach((group) => newGroups[group] = 1); - await db.set('groups', newGroups); - } - - // remove group entry + // Remove the group record after updating the `groups` record so that the state is consistent. await db.remove(`group:${groupID}`); }; @@ -86,22 +75,12 @@ exports.doesGroupExist = async (groupID) => { }; exports.createGroup = async () => { - // search for non existing groupID const groupID = `g.${randomString(16)}`; - - // create the group await db.set(`group:${groupID}`, {pads: {}}); - - // list the group - let groups = await exports.listAllGroups(); - groups = groups ? groups.groupIDs : []; - groups.push(groupID); - - // regenerate group list - const newGroups = {}; - groups.forEach((group) => newGroups[group] = 1); - await db.set('groups', newGroups); - + // Add the group to the `groups` record after the group's individual record is created so that + // the state is consistent. Note: UeberDB's setSub() method atomically reads the record, updates + // the appropriate property, and writes the result. + await db.setSub('groups', [groupID], 1); return {groupID}; }; From 777d045246635db68d7f83e220c367a5a2b2dd6f Mon Sep 17 00:00:00 2001 From: Richard Hansen Date: Sun, 28 Nov 2021 01:47:44 -0500 Subject: [PATCH 039/446] GroupManager: Clean up any mappings when deleting a group --- CHANGELOG.md | 2 ++ src/node/db/GroupManager.js | 28 ++++++++----------- .../backend/specs/api/sessionsAndGroups.js | 25 +++++++++++++++-- 3 files changed, 36 insertions(+), 19 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index a16f29c85..2643885d8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,6 +3,8 @@ ### Notable enhancements and fixes * Fixed a potential attribute pool corruption bug with `copyPadWithoutHistory`. +* Mappings created by the `createGroupIfNotExistsFor` HTTP API are now removed + from the database when the group is deleted. #### For plugin authors diff --git a/src/node/db/GroupManager.js b/src/node/db/GroupManager.js index de5efc2e4..29ab1b598 100644 --- a/src/node/db/GroupManager.js +++ b/src/node/db/GroupManager.js @@ -61,6 +61,7 @@ exports.deleteGroup = async (groupID) => { // writes the result. Setting a property to `undefined` deletes that property (JSON.stringify() // ignores such properties). db.setSub('groups', [groupID], undefined), + ...Object.keys(group.mappings || {}).map(async (m) => await db.remove(`mapper2group:${m}`)), ]); // Remove the group record after updating the `groups` record so that the state is consistent. @@ -76,7 +77,7 @@ exports.doesGroupExist = async (groupID) => { exports.createGroup = async () => { const groupID = `g.${randomString(16)}`; - await db.set(`group:${groupID}`, {pads: {}}); + await db.set(`group:${groupID}`, {pads: {}, mappings: {}}); // Add the group to the `groups` record after the group's individual record is created so that // the state is consistent. Note: UeberDB's setSub() method atomically reads the record, updates // the appropriate property, and writes the result. @@ -85,27 +86,20 @@ exports.createGroup = async () => { }; exports.createGroupIfNotExistsFor = async (groupMapper) => { - // ensure mapper is optional if (typeof groupMapper !== 'string') { throw new CustomError('groupMapper is not a string', 'apierror'); } - - // try to get a group for this mapper const groupID = await db.get(`mapper2group:${groupMapper}`); - - if (groupID) { - // there is a group for this mapper - const exists = await exports.doesGroupExist(groupID); - - if (exists) return {groupID}; - } - - // hah, the returned group doesn't exist, let's create one + if (groupID && await exports.doesGroupExist(groupID)) return {groupID}; const result = await exports.createGroup(); - - // create the mapper entry for this group - await db.set(`mapper2group:${groupMapper}`, result.groupID); - + await Promise.all([ + db.set(`mapper2group:${groupMapper}`, result.groupID), + // Remember the mapping in the group record so that it can be cleaned up when the group is + // deleted. Although the core Etherpad API does not support multiple mappings for the same + // group, the database record does support multiple mappings in case a plugin decides to extend + // the core Etherpad functionality. (It's also easy to implement it this way.) + db.setSub(`group:${result.groupID}`, ['mappings', groupMapper], 1), + ]); return result; }; diff --git a/src/tests/backend/specs/api/sessionsAndGroups.js b/src/tests/backend/specs/api/sessionsAndGroups.js index 83fdac698..eb181f01f 100644 --- a/src/tests/backend/specs/api/sessionsAndGroups.js +++ b/src/tests/backend/specs/api/sessionsAndGroups.js @@ -2,6 +2,7 @@ const assert = require('assert').strict; const common = require('../../common'); +const db = require('../../../../node/db/DB'); let agent; const apiKey = common.apiKey; @@ -89,13 +90,33 @@ describe(__filename, function () { }); it('createGroupIfNotExistsFor', async function () { - await agent.get(`${endPoint('createGroupIfNotExistsFor')}&groupMapper=management`) + const mapper = makeid(); + let groupId; + await agent.get(`${endPoint('createGroupIfNotExistsFor')}&groupMapper=${mapper}`) .expect(200) .expect('Content-Type', /json/) .expect((res) => { assert.equal(res.body.code, 0); - assert(res.body.data.groupID); + groupId = res.body.data.groupID; + assert(groupId); }); + // Passing the same mapper should return the same group ID. + await agent.get(`${endPoint('createGroupIfNotExistsFor')}&groupMapper=${mapper}`) + .expect(200) + .expect('Content-Type', /json/) + .expect((res) => { + assert.equal(res.body.code, 0); + assert.equal(res.body.data.groupID, groupId); + }); + // Deleting the group should clean up the mapping. + assert.equal(await db.get(`mapper2group:${mapper}`), groupId); + await agent.get(`${endPoint('deleteGroup')}&groupID=${groupId}`) + .expect(200) + .expect('Content-Type', /json/) + .expect((res) => { + assert.equal(res.body.code, 0); + }); + assert(await db.get(`mapper2group:${mapper}`) == null); }); // Test coverage for https://github.com/ether/etherpad-lite/issues/4227 From a02e45499deea04af0481fc4f51fb2f80b7bb78a Mon Sep 17 00:00:00 2001 From: Richard Hansen Date: Sun, 28 Nov 2021 23:39:15 -0500 Subject: [PATCH 040/446] Use the new AttributeMap and Changeset APIs --- src/node/db/Pad.js | 25 +++------------------ src/static/js/broadcast.js | 4 +--- src/tests/backend/specs/contentcollector.js | 8 ++----- 3 files changed, 6 insertions(+), 31 deletions(-) diff --git a/src/node/db/Pad.js b/src/node/db/Pad.js index 80ac5369d..0379f512a 100644 --- a/src/node/db/Pad.js +++ b/src/node/db/Pad.js @@ -3,7 +3,7 @@ * The pad object, defined with joose */ - +const AttributeMap = require('../../static/js/AttributeMap'); const Changeset = require('../../static/js/Changeset'); const ChatMessage = require('../../static/js/ChatMessage'); const AttributePool = require('../../static/js/AttributePool'); @@ -647,16 +647,6 @@ Pad.prototype.check = async function () { assert(pool instanceof AttributePool); await pool.check(); - const decodeAttribString = function* (str) { - const re = /\*([0-9a-z]+)|./gy; - let match; - while ((match = re.exec(str)) != null) { - const [m, n] = match; - if (n == null) throw new Error(`invalid character in attribute string: ${m}`); - yield Number.parseInt(n, 36); - } - }; - const authors = new Set(); pool.eachAttrib((k, v) => { if (k === 'author' && v) authors.add(v); @@ -681,9 +671,7 @@ Pad.prototype.check = async function () { Changeset.checkRep(changeset); const unpacked = Changeset.unpack(changeset); let text = atext.text; - const iter = Changeset.opIterator(unpacked.ops); - while (iter.hasNext()) { - const op = iter.next(); + for (const op of Changeset.deserializeOps(unpacked.ops)) { if (['=', '-'].includes(op.opcode)) { assert(text.length >= op.chars); const consumed = text.slice(0, op.chars); @@ -692,14 +680,7 @@ Pad.prototype.check = async function () { if (op.lines > 0) assert(consumed.endsWith('\n')); text = text.slice(op.chars); } - let prevK = null; - for (const n of decodeAttribString(op.attribs)) { - const attrib = pool.getAttrib(n); - assert(attrib != null); - const [k] = attrib; - assert(prevK == null || prevK < k); - prevK = k; - } + assert.equal(op.attribs, AttributeMap.fromString(op.attribs, pool).toString()); } atext = Changeset.applyToAText(changeset, atext, pool); assert.deepEqual(await this.getInternalRevisionAText(r), atext); diff --git a/src/static/js/broadcast.js b/src/static/js/broadcast.js index 4014d5829..cd2211ae1 100644 --- a/src/static/js/broadcast.js +++ b/src/static/js/broadcast.js @@ -117,9 +117,7 @@ const loadBroadcastJS = (socket, sendSocketMsg, fireWhenAllScriptsAreLoaded, Bro getActiveAuthors() { const authorIds = new Set(); for (const aline of this.alines) { - const opIter = Changeset.opIterator(aline); - while (opIter.hasNext()) { - const op = opIter.next(); + for (const op of Changeset.deserializeOps(aline)) { for (const [k, v] of attributes.attribsFromString(op.attribs, this.apool)) { if (k !== 'author') continue; if (v) authorIds.add(v); diff --git a/src/tests/backend/specs/contentcollector.js b/src/tests/backend/specs/contentcollector.js index c2cee5338..a4696307e 100644 --- a/src/tests/backend/specs/contentcollector.js +++ b/src/tests/backend/specs/contentcollector.js @@ -346,9 +346,7 @@ describe(__filename, function () { // numbers do not change if the attribute processing code changes.) for (const attrib of knownAttribs) apool.putAttrib(attrib); for (const aline of tc.wantAlines) { - const opIter = Changeset.opIterator(aline); - while (opIter.hasNext()) { - const op = opIter.next(); + for (const op of Changeset.deserializeOps(aline)) { for (const n of attributes.decodeAttribString(op.attribs)) { assert(n < knownAttribs.length); } @@ -375,9 +373,7 @@ describe(__filename, function () { gotAttribs.push(gotAlineAttribs); const wantAlineAttribs = []; wantAttribs.push(wantAlineAttribs); - const opIter = Changeset.opIterator(aline); - while (opIter.hasNext()) { - const op = opIter.next(); + for (const op of Changeset.deserializeOps(aline)) { const gotOpAttribs = [...attributes.attribsFromString(op.attribs, apool)]; gotAlineAttribs.push(gotOpAttribs); wantAlineAttribs.push(attributes.sort([...gotOpAttribs])); From 48080411fcb9d2514fa551202b630e285e36d1c1 Mon Sep 17 00:00:00 2001 From: Tommy Date: Mon, 8 Nov 2021 11:40:34 -0500 Subject: [PATCH 041/446] Docker: Update to the latest LTS image The Node.js 14 slim image has quite a few vulnerabilities, and I have tested the latest slim image. It works just fine. When installing plugins, `--legacy-peer-deps` is passed to npm because npm v7 (which comes with Node.js v16, the current LTS) changed how peer deps are handled. The new behavior is incompatible with how plugins have historically been installed. --- Dockerfile | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/Dockerfile b/Dockerfile index c6339ae72..8cf8f04c5 100644 --- a/Dockerfile +++ b/Dockerfile @@ -4,7 +4,7 @@ # # Author: muxator -FROM node:14-buster-slim +FROM node:lts-slim LABEL maintainer="Etherpad team, https://github.com/ether/etherpad-lite" # plugins to install while building the container. By default no plugins are @@ -85,7 +85,7 @@ COPY --chown=etherpad:etherpad ./ ./ # seems to confuse tools such as `npm outdated`, `npm update`, and some ESLint # rules. RUN { [ -z "${ETHERPAD_PLUGINS}" ] || \ - npm install --no-save ${ETHERPAD_PLUGINS}; } && \ + npm install --no-save --legacy-peer-deps ${ETHERPAD_PLUGINS}; } && \ src/bin/installDeps.sh && \ rm -rf ~/.npm From 306e46c21d74822a6cfca185cd200ea11fc0f9ff Mon Sep 17 00:00:00 2001 From: Richard Hansen Date: Sun, 28 Nov 2021 02:24:52 -0500 Subject: [PATCH 042/446] Docker: Upgrade Debian packages --- Dockerfile | 1 + 1 file changed, 1 insertion(+) diff --git a/Dockerfile b/Dockerfile index 8cf8f04c5..34c16d165 100644 --- a/Dockerfile +++ b/Dockerfile @@ -60,6 +60,7 @@ RUN mkdir -p "${EP_DIR}" && chown etherpad:etherpad "${EP_DIR}" RUN export DEBIAN_FRONTEND=noninteractive; \ mkdir -p /usr/share/man/man1 && \ apt-get -qq update && \ + apt-get -qq dist-upgrade && \ apt-get -qq --no-install-recommends install \ ca-certificates \ git \ From df459c1278d70354de41d890dcff8933c70b4662 Mon Sep 17 00:00:00 2001 From: Richard Hansen Date: Mon, 29 Nov 2021 20:35:29 -0500 Subject: [PATCH 043/446] Enable Dependabot for GitHub Actions --- .github/dependabot.yml | 7 +++++++ 1 file changed, 7 insertions(+) create mode 100644 .github/dependabot.yml diff --git a/.github/dependabot.yml b/.github/dependabot.yml new file mode 100644 index 000000000..2c7d17083 --- /dev/null +++ b/.github/dependabot.yml @@ -0,0 +1,7 @@ +version: 2 +updates: + # Maintain dependencies for GitHub Actions + - package-ecosystem: "github-actions" + directory: "/" + schedule: + interval: "daily" From 40854b0cfdb8b07a06d7ae8b789ff90f1cbf8d47 Mon Sep 17 00:00:00 2001 From: Richard Hansen Date: Mon, 29 Nov 2021 21:02:41 -0500 Subject: [PATCH 044/446] GitHub workflow to build and publish Docker images --- .github/workflows/docker.yml | 36 ++++++++++++++++++++++++++++++++++++ 1 file changed, 36 insertions(+) create mode 100644 .github/workflows/docker.yml diff --git a/.github/workflows/docker.yml b/.github/workflows/docker.yml new file mode 100644 index 000000000..be0d2469a --- /dev/null +++ b/.github/workflows/docker.yml @@ -0,0 +1,36 @@ +name: Docker +on: + push: + branches: + - 'develop' + tags: + - 'v?[0-9]+.[0-9]+.[0-9]+' +jobs: + docker: + runs-on: ubuntu-latest + steps: + - name: Check out + uses: actions/checkout@v2 + - name: Docker meta + id: meta + uses: docker/metadata-action@v3 + with: + images: etherpad/etherpad + tags: | + type=ref,event=branch + type=semver,pattern={{version}} + type=semver,pattern={{major}}.{{minor}} + type=semver,pattern={{major}} + - name: Log in to Docker Hub + if: github.event_name != 'pull_request' + uses: docker/login-action@v1 + with: + username: ${{ secrets.DOCKERHUB_USERNAME }} + password: ${{ secrets.DOCKERHUB_TOKEN }} + - name: Build and push + uses: docker/build-push-action@v2 + with: + context: . + push: ${{ github.event_name != 'pull_request' }} + tags: ${{ steps.meta.outputs.tags }} + labels: ${{ steps.meta.outputs.labels }} From 68933718f62691094cae978cd84e5dd05e64f313 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Tue, 30 Nov 2021 02:05:07 +0000 Subject: [PATCH 045/446] Bump joncloud/makensis-action from 3.4 to 3.6 Bumps [joncloud/makensis-action](https://github.com/joncloud/makensis-action) from 3.4 to 3.6. - [Release notes](https://github.com/joncloud/makensis-action/releases) - [Commits](https://github.com/joncloud/makensis-action/compare/v3.4...v3.6) --- updated-dependencies: - dependency-name: joncloud/makensis-action dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] --- .github/workflows/windows-installer.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/windows-installer.yml b/.github/workflows/windows-installer.yml index 37d86a9a6..b369474db 100644 --- a/.github/workflows/windows-installer.yml +++ b/.github/workflows/windows-installer.yml @@ -47,7 +47,7 @@ jobs: run: git clone https://github.com/ether/etherpad_nsis.git - name: Create installer - uses: joncloud/makensis-action@v3.4 + uses: joncloud/makensis-action@v3.6 with: script-file: 'etherpad_nsis/etherpad.nsi' From e4944b8bfae394555bef6b5c2a08f7a7d4f19c22 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Tue, 30 Nov 2021 02:05:11 +0000 Subject: [PATCH 046/446] Bump saucelabs/sauce-connect-action from 1.1.2 to 2.0.0 Bumps [saucelabs/sauce-connect-action](https://github.com/saucelabs/sauce-connect-action) from 1.1.2 to 2.0.0. - [Release notes](https://github.com/saucelabs/sauce-connect-action/releases) - [Changelog](https://github.com/saucelabs/sauce-connect-action/blob/main/CHANGELOG.md) - [Commits](https://github.com/saucelabs/sauce-connect-action/compare/v1.1.2...v2.0.0) --- updated-dependencies: - dependency-name: saucelabs/sauce-connect-action dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] --- .github/workflows/frontend-admin-tests.yml | 2 +- .github/workflows/frontend-tests.yml | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/.github/workflows/frontend-admin-tests.yml b/.github/workflows/frontend-admin-tests.yml index 5fb0f39c2..00c390717 100644 --- a/.github/workflows/frontend-admin-tests.yml +++ b/.github/workflows/frontend-admin-tests.yml @@ -63,7 +63,7 @@ jobs: - name: Remove standard frontend test files, so only admin tests are run run: mv src/tests/frontend/specs/* /tmp && mv /tmp/admin*.js src/tests/frontend/specs - - uses: saucelabs/sauce-connect-action@v1.1.2 + - uses: saucelabs/sauce-connect-action@v2.0.0 with: username: ${{ secrets.SAUCE_USERNAME }} accessKey: ${{ secrets.SAUCE_ACCESS_KEY }} diff --git a/.github/workflows/frontend-tests.yml b/.github/workflows/frontend-tests.yml index bc138416c..e8b657a39 100644 --- a/.github/workflows/frontend-tests.yml +++ b/.github/workflows/frontend-tests.yml @@ -36,7 +36,7 @@ jobs: run: | sed -e '/^ *"importExportRateLimiting":/,/^ *\}/ s/"max":.*/"max": 0/' -i settings.json - - uses: saucelabs/sauce-connect-action@v1 + - uses: saucelabs/sauce-connect-action@v2.0.0 with: username: ${{ secrets.SAUCE_USERNAME }} accessKey: ${{ secrets.SAUCE_ACCESS_KEY }} @@ -116,7 +116,7 @@ jobs: - name: Remove standard frontend test files, so only plugin tests are run run: rm src/tests/frontend/specs/* - - uses: saucelabs/sauce-connect-action@v1 + - uses: saucelabs/sauce-connect-action@v2.0.0 with: username: ${{ secrets.SAUCE_USERNAME }} accessKey: ${{ secrets.SAUCE_ACCESS_KEY }} From c4f18a9b3a6d9819a034ba808108a599f332a42b Mon Sep 17 00:00:00 2001 From: Richard Hansen Date: Tue, 30 Nov 2021 23:11:21 -0500 Subject: [PATCH 047/446] padutils: Rename `warnWithStack()` to `warnDeprecated()` This makes it more legitimate for tests to disable the warnings when testing deprecated functionality. --- src/static/js/Changeset.js | 23 ++++++++++++----------- src/static/js/pad_utils.js | 4 ++-- 2 files changed, 14 insertions(+), 13 deletions(-) diff --git a/src/static/js/Changeset.js b/src/static/js/Changeset.js index 0779884c0..ce8e8a789 100644 --- a/src/static/js/Changeset.js +++ b/src/static/js/Changeset.js @@ -253,7 +253,7 @@ class OpIter { * @returns {OpIter} Operator iterator object. */ exports.opIterator = (opsStr) => { - padutils.warnWithStack( + padutils.warnDeprecated( 'Changeset.opIterator() is deprecated; use Changeset.deserializeOps() instead'); return new OpIter(opsStr); }; @@ -278,7 +278,7 @@ const clearOp = (op) => { * @returns {Op} */ exports.newOp = (optOpcode) => { - padutils.warnWithStack('Changeset.newOp() is deprecated; use the Changeset.Op class instead'); + padutils.warnDeprecated('Changeset.newOp() is deprecated; use the Changeset.Op class instead'); return new Op(optOpcode); }; @@ -477,8 +477,8 @@ exports.smartOpAssembler = () => { * attribute key, value pairs. */ const appendOpWithText = (opcode, text, attribs, pool) => { - padutils.warnWithStack('Changeset.smartOpAssembler().appendOpWithText() is deprecated; ' + - 'use opsFromText() instead.'); + padutils.warnDeprecated('Changeset.smartOpAssembler().appendOpWithText() is deprecated; ' + + 'use opsFromText() instead.'); for (const op of opsFromText(opcode, text, attribs, pool)) append(op); }; @@ -1647,8 +1647,8 @@ exports.makeAttribution = (text) => { * @param {Function} func - function to call */ exports.eachAttribNumber = (cs, func) => { - padutils.warnWithStack('Changeset.eachAttribNumber() is deprecated; ' + - 'use attributes.decodeAttribString() instead'); + padutils.warnDeprecated( + 'Changeset.eachAttribNumber() is deprecated; use attributes.decodeAttribString() instead'); let dollarPos = cs.indexOf('$'); if (dollarPos < 0) { dollarPos = cs.length; @@ -1801,7 +1801,7 @@ exports.opsFromAText = function* (atext) { * @param assem - Assembler like SmartOpAssembler TODO add desc */ exports.appendATextToAssembler = (atext, assem) => { - padutils.warnWithStack( + padutils.warnDeprecated( 'Changeset.appendATextToAssembler() is deprecated; use Changeset.opsFromAText() instead'); for (const op of exports.opsFromAText(atext)) assem.append(op); }; @@ -1854,7 +1854,8 @@ const attribsAttributeValue = (attribs, key, pool) => { * @returns {string} */ exports.opAttributeValue = (op, key, pool) => { - padutils.warnWithStack('Changeset.opAttributeValue() is deprecated; use an AttributeMap instead'); + padutils.warnDeprecated( + 'Changeset.opAttributeValue() is deprecated; use an AttributeMap instead'); return attribsAttributeValue(op.attribs, key, pool); }; @@ -1868,8 +1869,8 @@ exports.opAttributeValue = (op, key, pool) => { * @returns {string} */ exports.attribsAttributeValue = (attribs, key, pool) => { - padutils.warnWithStack('Changeset.attribsAttributeValue() is deprecated; ' + - 'use an AttributeMap instead'); + padutils.warnDeprecated( + 'Changeset.attribsAttributeValue() is deprecated; use an AttributeMap instead'); return attribsAttributeValue(attribs, key, pool); }; @@ -1980,7 +1981,7 @@ exports.builder = (oldLen) => { * @returns {AttributeString} */ exports.makeAttribsString = (opcode, attribs, pool) => { - padutils.warnWithStack( + padutils.warnDeprecated( 'Changeset.makeAttribsString() is deprecated; ' + 'use AttributeMap.prototype.toString() or attributes.attribsToString() instead'); if (!attribs || !['=', '+'].includes(opcode)) return ''; diff --git a/src/static/js/pad_utils.js b/src/static/js/pad_utils.js index 9bea959da..27190c3d1 100644 --- a/src/static/js/pad_utils.js +++ b/src/static/js/pad_utils.js @@ -100,9 +100,9 @@ const padutils = { * * @param {...*} args - Passed to `console.warn`, with a stack trace appended. */ - warnWithStack: (...args) => { + warnDeprecated: (...args) => { const err = new Error(); - if (Error.captureStackTrace) Error.captureStackTrace(err, padutils.warnWithStack); + if (Error.captureStackTrace) Error.captureStackTrace(err, padutils.warnDeprecated); err.name = ''; if (err.stack) args.push(err.stack); console.warn(...args); From 6beb5dcaf534281b87d87933e2e231d627df108d Mon Sep 17 00:00:00 2001 From: Richard Hansen Date: Tue, 30 Nov 2021 23:17:35 -0500 Subject: [PATCH 048/446] tests: Disable deprecation warnings when testing deprecated functions --- src/static/js/pad_utils.js | 1 + src/tests/frontend/specs/easysync-other.js | 30 ++++++++++++++-------- 2 files changed, 21 insertions(+), 10 deletions(-) diff --git a/src/static/js/pad_utils.js b/src/static/js/pad_utils.js index 27190c3d1..9d5367023 100644 --- a/src/static/js/pad_utils.js +++ b/src/static/js/pad_utils.js @@ -101,6 +101,7 @@ const padutils = { * @param {...*} args - Passed to `console.warn`, with a stack trace appended. */ warnDeprecated: (...args) => { + if (padutils.warnDeprecated.disabledForTestingOnly) return; const err = new Error(); if (Error.captureStackTrace) Error.captureStackTrace(err, padutils.warnDeprecated); err.name = ''; diff --git a/src/tests/frontend/specs/easysync-other.js b/src/tests/frontend/specs/easysync-other.js index 26376713a..856b7df62 100644 --- a/src/tests/frontend/specs/easysync-other.js +++ b/src/tests/frontend/specs/easysync-other.js @@ -3,6 +3,7 @@ const Changeset = require('../../../static/js/Changeset'); const AttributePool = require('../../../static/js/AttributePool'); const {randomMultiline, poolOrArray} = require('../easysync-helper.js'); +const {padutils} = require('../../../static/js/pad_utils'); describe('easysync-other', function () { describe('filter attribute numbers', function () { @@ -23,8 +24,12 @@ describe('easysync-other', function () { const testMakeAttribsString = (testId, pool, opcode, attribs, correctString) => { it(`testMakeAttribsString#${testId}`, async function () { const p = poolOrArray(pool); - const str = Changeset.makeAttribsString(opcode, attribs, p); - expect(str).to.equal(correctString); + padutils.warnDeprecated.disabledForTestingOnly = true; + try { + expect(Changeset.makeAttribsString(opcode, attribs, p)).to.equal(correctString); + } finally { + delete padutils.warnDeprecated.disabledForTestingOnly; + } }); }; @@ -81,14 +86,19 @@ describe('easysync-other', function () { const stringOp = (str) => Changeset.deserializeOps(str).next().value; - expect(Changeset.opAttributeValue(stringOp('*0*1+1'), 'name', p)).to.equal('david'); - expect(Changeset.opAttributeValue(stringOp('*0+1'), 'name', p)).to.equal('david'); - expect(Changeset.opAttributeValue(stringOp('*1+1'), 'name', p)).to.equal(''); - expect(Changeset.opAttributeValue(stringOp('+1'), 'name', p)).to.equal(''); - expect(Changeset.opAttributeValue(stringOp('*0*1+1'), 'color', p)).to.equal('green'); - expect(Changeset.opAttributeValue(stringOp('*1+1'), 'color', p)).to.equal('green'); - expect(Changeset.opAttributeValue(stringOp('*0+1'), 'color', p)).to.equal(''); - expect(Changeset.opAttributeValue(stringOp('+1'), 'color', p)).to.equal(''); + padutils.warnDeprecated.disabledForTestingOnly = true; + try { + expect(Changeset.opAttributeValue(stringOp('*0*1+1'), 'name', p)).to.equal('david'); + expect(Changeset.opAttributeValue(stringOp('*0+1'), 'name', p)).to.equal('david'); + expect(Changeset.opAttributeValue(stringOp('*1+1'), 'name', p)).to.equal(''); + expect(Changeset.opAttributeValue(stringOp('+1'), 'name', p)).to.equal(''); + expect(Changeset.opAttributeValue(stringOp('*0*1+1'), 'color', p)).to.equal('green'); + expect(Changeset.opAttributeValue(stringOp('*1+1'), 'color', p)).to.equal('green'); + expect(Changeset.opAttributeValue(stringOp('*0+1'), 'color', p)).to.equal(''); + expect(Changeset.opAttributeValue(stringOp('+1'), 'color', p)).to.equal(''); + } finally { + delete padutils.warnDeprecated.disabledForTestingOnly; + } }); describe('applyToAttribution', function () { From f4257a28ba1fcee5a053e1ce14a6ea771f54b46d Mon Sep 17 00:00:00 2001 From: Richard Hansen Date: Sat, 4 Dec 2021 21:04:22 -0500 Subject: [PATCH 049/446] pad: Delete duplicate `decodeURIComponent()` calls `URL.searchParams` already decodes the value. Also delete some useless comments. --- src/static/js/pad.js | 12 ++++-------- 1 file changed, 4 insertions(+), 8 deletions(-) diff --git a/src/static/js/pad.js b/src/static/js/pad.js index 306c2b191..3a372d1ce 100644 --- a/src/static/js/pad.js +++ b/src/static/js/pad.js @@ -95,24 +95,20 @@ const getParameters = [ settings.useMonospaceFontGlobal = true; }, }, - // If the username is set as a parameter we should set a global value that we can call once we - // have initiated the pad. { name: 'userName', checkVal: null, callback: (val) => { - settings.globalUserName = decodeURIComponent(val); - clientVars.userName = decodeURIComponent(val); + settings.globalUserName = val; + clientVars.userName = val; }, }, - // If the userColor is set as a parameter, set a global value to use once we have initiated the - // pad. { name: 'userColor', checkVal: null, callback: (val) => { - settings.globalUserColor = decodeURIComponent(val); - clientVars.userColor = decodeURIComponent(val); + settings.globalUserColor = val; + clientVars.userColor = val; }, }, { From 7ff71cd41e60012026833e14f32c257e81be05b8 Mon Sep 17 00:00:00 2001 From: Richard Hansen Date: Sat, 4 Dec 2021 21:08:17 -0500 Subject: [PATCH 050/446] pad: Ignore `null` values in `padOptions` from `settings.json` --- src/static/js/pad.js | 1 + 1 file changed, 1 insertion(+) diff --git a/src/static/js/pad.js b/src/static/js/pad.js index 3a372d1ce..b7ecdd199 100644 --- a/src/static/js/pad.js +++ b/src/static/js/pad.js @@ -146,6 +146,7 @@ const getParams = () => { // Tries server enforced options first.. for (const setting of getParameters) { const value = clientVars.padOptions[setting.name]; + if (value == null) continue; if (value.toString() === setting.checkVal) { setting.callback(value); } From 61b608e2645c7442b82e1a4b873399812e1da18c Mon Sep 17 00:00:00 2001 From: Richard Hansen Date: Sat, 4 Dec 2021 21:33:38 -0500 Subject: [PATCH 051/446] pad: Use `null` as default for `lang` option It doesn't make sense to override the browser's language with `en-gb` by default. Note that this change has no effect due to a bug in how pad options are processed; that bug will be fixed in a future commit. --- doc/docker.md | 2 +- settings.json.docker | 2 +- settings.json.template | 2 +- src/node/utils/Settings.js | 2 +- 4 files changed, 4 insertions(+), 4 deletions(-) diff --git a/doc/docker.md b/doc/docker.md index 91caab97e..238b2cfba 100644 --- a/doc/docker.md +++ b/doc/docker.md @@ -121,7 +121,7 @@ If your database needs additional settings, you will have to use a personalized | `PAD_OPTIONS_RTL` | | `false` | | `PAD_OPTIONS_ALWAYS_SHOW_CHAT` | | `false` | | `PAD_OPTIONS_CHAT_AND_USERS` | | `false` | -| `PAD_OPTIONS_LANG` | | `en-gb` | +| `PAD_OPTIONS_LANG` | | `null` | ### Shortcuts diff --git a/settings.json.docker b/settings.json.docker index ed1be901d..338bfc951 100644 --- a/settings.json.docker +++ b/settings.json.docker @@ -239,7 +239,7 @@ "rtl": "${PAD_OPTIONS_RTL:false}", "alwaysShowChat": "${PAD_OPTIONS_ALWAYS_SHOW_CHAT:false}", "chatAndUsers": "${PAD_OPTIONS_CHAT_AND_USERS:false}", - "lang": "${PAD_OPTIONS_LANG:en-gb}" + "lang": "${PAD_OPTIONS_LANG:null}" }, /* diff --git a/settings.json.template b/settings.json.template index 8b8766be8..9bb040917 100644 --- a/settings.json.template +++ b/settings.json.template @@ -240,7 +240,7 @@ "rtl": false, "alwaysShowChat": false, "chatAndUsers": false, - "lang": "en-gb" + "lang": null }, /* diff --git a/src/node/utils/Settings.js b/src/node/utils/Settings.js index 814601f89..4beba3962 100644 --- a/src/node/utils/Settings.js +++ b/src/node/utils/Settings.js @@ -170,7 +170,7 @@ exports.padOptions = { rtl: false, alwaysShowChat: false, chatAndUsers: false, - lang: 'en-gb', + lang: null, }; /** From 8c857a85ac9af08ebec1c94031ca7c54bed34c38 Mon Sep 17 00:00:00 2001 From: Richard Hansen Date: Sat, 4 Dec 2021 21:36:02 -0500 Subject: [PATCH 052/446] pad: Use `null` as default for `userName`, `userColor` options These options are used as strings, so it doesn't make sense to default them to a boolean value. Note that this change has no effect due to a bug in how pad options are processed; that bug will be fixed in a future commit. --- doc/docker.md | 4 ++-- settings.json.docker | 4 ++-- settings.json.template | 4 ++-- src/node/utils/Settings.js | 4 ++-- 4 files changed, 8 insertions(+), 8 deletions(-) diff --git a/doc/docker.md b/doc/docker.md index 238b2cfba..b6697d75a 100644 --- a/doc/docker.md +++ b/doc/docker.md @@ -116,8 +116,8 @@ If your database needs additional settings, you will have to use a personalized | `PAD_OPTIONS_SHOW_CHAT` | | `true` | | `PAD_OPTIONS_SHOW_LINE_NUMBERS` | | `true` | | `PAD_OPTIONS_USE_MONOSPACE_FONT` | | `false` | -| `PAD_OPTIONS_USER_NAME` | | `false` | -| `PAD_OPTIONS_USER_COLOR` | | `false` | +| `PAD_OPTIONS_USER_NAME` | | `null` | +| `PAD_OPTIONS_USER_COLOR` | | `null` | | `PAD_OPTIONS_RTL` | | `false` | | `PAD_OPTIONS_ALWAYS_SHOW_CHAT` | | `false` | | `PAD_OPTIONS_CHAT_AND_USERS` | | `false` | diff --git a/settings.json.docker b/settings.json.docker index 338bfc951..1642579e1 100644 --- a/settings.json.docker +++ b/settings.json.docker @@ -234,8 +234,8 @@ "showChat": "${PAD_OPTIONS_SHOW_CHAT:true}", "showLineNumbers": "${PAD_OPTIONS_SHOW_LINE_NUMBERS:true}", "useMonospaceFont": "${PAD_OPTIONS_USE_MONOSPACE_FONT:false}", - "userName": "${PAD_OPTIONS_USER_NAME:false}", - "userColor": "${PAD_OPTIONS_USER_COLOR:false}", + "userName": "${PAD_OPTIONS_USER_NAME:null}", + "userColor": "${PAD_OPTIONS_USER_COLOR:null}", "rtl": "${PAD_OPTIONS_RTL:false}", "alwaysShowChat": "${PAD_OPTIONS_ALWAYS_SHOW_CHAT:false}", "chatAndUsers": "${PAD_OPTIONS_CHAT_AND_USERS:false}", diff --git a/settings.json.template b/settings.json.template index 9bb040917..b802248f1 100644 --- a/settings.json.template +++ b/settings.json.template @@ -235,8 +235,8 @@ "showChat": true, "showLineNumbers": true, "useMonospaceFont": false, - "userName": false, - "userColor": false, + "userName": null, + "userColor": null, "rtl": false, "alwaysShowChat": false, "chatAndUsers": false, diff --git a/src/node/utils/Settings.js b/src/node/utils/Settings.js index 4beba3962..42d29edb8 100644 --- a/src/node/utils/Settings.js +++ b/src/node/utils/Settings.js @@ -165,8 +165,8 @@ exports.padOptions = { showChat: true, showLineNumbers: true, useMonospaceFont: false, - userName: false, - userColor: false, + userName: null, + userColor: null, rtl: false, alwaysShowChat: false, chatAndUsers: false, From f8b4189bc451ebc2a80c4f13e34bf4537570a7da Mon Sep 17 00:00:00 2001 From: Richard Hansen Date: Sat, 4 Dec 2021 22:35:40 -0500 Subject: [PATCH 053/446] pad: Always pass strings to pad option callbacks --- src/static/js/pad.js | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/src/static/js/pad.js b/src/static/js/pad.js index b7ecdd199..bba8008f2 100644 --- a/src/static/js/pad.js +++ b/src/static/js/pad.js @@ -145,9 +145,10 @@ const getParameters = [ const getParams = () => { // Tries server enforced options first.. for (const setting of getParameters) { - const value = clientVars.padOptions[setting.name]; + let value = clientVars.padOptions[setting.name]; if (value == null) continue; - if (value.toString() === setting.checkVal) { + value = value.toString(); + if (value === setting.checkVal) { setting.callback(value); } } From 99fae2ec6e180dbefb6e8c048e83e9e6b7a2371b Mon Sep 17 00:00:00 2001 From: Richard Hansen Date: Sat, 4 Dec 2021 21:38:48 -0500 Subject: [PATCH 054/446] pad: Fix application of `padOptions` values from `settings.json` --- CHANGELOG.md | 6 ++++++ src/static/js/pad.js | 2 +- 2 files changed, 7 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 5ed2d637c..7b9efcb01 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,12 @@ ### Notable enhancements and fixes +* The following settings from `settings.json` are now applied as expected (they + were unintentionally ignored before): + * `padOptions.lang` + * `padOptions.showChat` + * `padOptions.userColor` + * `padOptions.userName` * Fixed a potential attribute pool corruption bug with `copyPadWithoutHistory`. * Mappings created by the `createGroupIfNotExistsFor` HTTP API are now removed from the database when the group is deleted. diff --git a/src/static/js/pad.js b/src/static/js/pad.js index bba8008f2..1696acc56 100644 --- a/src/static/js/pad.js +++ b/src/static/js/pad.js @@ -148,7 +148,7 @@ const getParams = () => { let value = clientVars.padOptions[setting.name]; if (value == null) continue; value = value.toString(); - if (value === setting.checkVal) { + if (value === setting.checkVal || setting.checkVal == null) { setting.callback(value); } } From 75ee1ef5352f4f88472c71c6a23237a4291468a8 Mon Sep 17 00:00:00 2001 From: Richard Hansen Date: Sun, 5 Dec 2021 17:33:59 -0500 Subject: [PATCH 055/446] Docker: Add `.git/rr-cache/` to `.dockerignore` --- .dockerignore | 1 + 1 file changed, 1 insertion(+) diff --git a/.dockerignore b/.dockerignore index 28c6753f5..d8d3a3ebe 100644 --- a/.dockerignore +++ b/.dockerignore @@ -19,6 +19,7 @@ Dockerfile .git/ORIG_HEAD .git/packed-refs .git/refs/remotes/ +.git/rr-cache/ .gitignore settings.json From 6cca27dea6a98acf7b487c52df3e8f6821d39ac7 Mon Sep 17 00:00:00 2001 From: John McLear Date: Fri, 24 Sep 2021 15:28:19 +0100 Subject: [PATCH 056/446] API: `getText` with old revision should only return text, not atext Co-authored-by: Richard Hansen --- CHANGELOG.md | 2 ++ src/node/db/API.js | 4 +++- src/tests/backend/specs/api/pad.js | 17 +++++++++++++++++ 3 files changed, 22 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 7b9efcb01..2b41449a8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,8 @@ * `padOptions.showChat` * `padOptions.userColor` * `padOptions.userName` +* Fixed the return value of the `getText` HTTP API when called with a specific + revision. * Fixed a potential attribute pool corruption bug with `copyPadWithoutHistory`. * Mappings created by the `createGroupIfNotExistsFor` HTTP API are now removed from the database when the group is deleted. diff --git a/src/node/db/API.js b/src/node/db/API.js index 0c3ddae46..49d6b9cef 100644 --- a/src/node/db/API.js +++ b/src/node/db/API.js @@ -172,7 +172,9 @@ exports.getText = async (padID, rev) => { } // get the text of this revision - const text = await pad.getInternalRevisionAText(rev); + // getInternalRevisionAText() returns an atext object but we only want the .text inside it. + // Details at https://github.com/ether/etherpad-lite/issues/5073 + const {text} = await pad.getInternalRevisionAText(rev); return {text}; } diff --git a/src/tests/backend/specs/api/pad.js b/src/tests/backend/specs/api/pad.js index 7aab137b9..41c30b8a0 100644 --- a/src/tests/backend/specs/api/pad.js +++ b/src/tests/backend/specs/api/pad.js @@ -102,6 +102,7 @@ describe(__filename, function () { -> getLastEdited(padID) -- Should not be 0 -> appendText(padID, "hello") -> getText(padID) -- Should be "hello worldhello" + -> getText(padID, rev=2) - should return "hello world" -> setHTML(padID) -- Should fail on invalid HTML -> setHTML(padID) *3 -- Should fail on invalid HTML -> getHTML(padID) -- Should return HTML close to posted HTML @@ -401,6 +402,22 @@ describe(__filename, function () { assert.equal(res.body.data.text, `${text}hello\n`); }); + it('getText of old revision', async function () { + let res = await agent.get(`${endPoint('getRevisionsCount')}&padID=${testPadId}`) + .expect(200) + .expect('Content-Type', /json/); + assert.equal(res.body.code, 0); + const rev = res.body.data.revisions; + assert(rev != null); + assert(Number.isInteger(rev)); + assert(rev > 0); + res = await agent.get(`${endPoint('getText')}&padID=${testPadId}&rev=${rev - 1}`) + .expect(200) + .expect('Content-Type', /json/); + assert.equal(res.body.code, 0); + assert.equal(res.body.data.text, `${text}\n`); + }); + it('Sets the HTML of a Pad attempting to pass ugly HTML', async function () { const html = '
    Hello HTML
    '; const res = await agent.post(endPoint('setHTML')) From 841bc10039e8da4ee3d295e632474ecf31f51f3e Mon Sep 17 00:00:00 2001 From: "translatewiki.net" Date: Mon, 6 Dec 2021 13:03:04 +0100 Subject: [PATCH 057/446] Localisation updates from https://translatewiki.net. --- src/locales/it.json | 1 + 1 file changed, 1 insertion(+) diff --git a/src/locales/it.json b/src/locales/it.json index cd3af6a97..23f7f42ce 100644 --- a/src/locales/it.json +++ b/src/locales/it.json @@ -17,6 +17,7 @@ "admin_plugins.name": "Nome", "admin_plugins.version": "Versione", "admin_settings": "Impostazioni", + "admin_settings.current_save.value": "Salva impostazioni", "index.newPad": "Nuovo pad", "index.createOpenPad": "o crea/apre un pad con il nome:", "index.openPad": "apri un Pad esistente col nome:", From d1da8f1ebdfd316affefc48d79f427e2d00b8842 Mon Sep 17 00:00:00 2001 From: "translatewiki.net" Date: Thu, 9 Dec 2021 13:03:48 +0100 Subject: [PATCH 058/446] Localisation updates from https://translatewiki.net. --- src/locales/es.json | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/src/locales/es.json b/src/locales/es.json index 33e80b425..5b7497315 100644 --- a/src/locales/es.json +++ b/src/locales/es.json @@ -2,6 +2,7 @@ "@metadata": { "authors": [ "Armando-Martin", + "DDPAT", "Dgstranz", "Fitoschido", "Jacobo", @@ -98,7 +99,7 @@ "pad.importExport.exportopen": "ODF (Open Document Format)", "pad.importExport.abiword.innerHTML": "Solo es posible importar texto sin formato o en HTML. Para obtener funciones de importación más avanzadas es necesario instalar AbiWord.", "pad.modals.connected": "Conectado.", - "pad.modals.reconnecting": "Reconectando a tu pad..", + "pad.modals.reconnecting": "Reconectando a tu pad...", "pad.modals.forcereconnect": "Forzar reconexión", "pad.modals.reconnecttimer": "Se intentará reconectar en", "pad.modals.cancel": "Cancelar", From 4d457f629638f1b00b2840b52a97d652ddb6c9cb Mon Sep 17 00:00:00 2001 From: Richard Hansen Date: Fri, 10 Dec 2021 02:34:13 -0500 Subject: [PATCH 059/446] ImportHandler: Pass `ImportError` to `import` hook --- CHANGELOG.md | 1 + doc/api/hooks_server-side.md | 13 +++++++++++++ src/node/handler/ImportHandler.js | 4 ++-- 3 files changed, 16 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 2b41449a8..8d044260d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -19,6 +19,7 @@ * New APIs for processing attributes: `ep_etherpad-lite/static/js/attributes` (low-level API) and `ep_etherpad-lite/static/js/AttributeMap` (high-level API). +* The `import` server-side hook has a new `ImportError` context property. ### Compatibility changes diff --git a/doc/api/hooks_server-side.md b/doc/api/hooks_server-side.md index 34a906ca6..f7aa6b0b6 100644 --- a/doc/api/hooks_server-side.md +++ b/doc/api/hooks_server-side.md @@ -821,6 +821,19 @@ Context properties: period** (examples: `'.docx'`, `'.html'`, `'.etherpad'`). * `padId`: The identifier of the destination pad. * `srcFile`: The document to convert. +* `ImportError`: Subclass of Error that can be thrown to provide a specific + error message to the user. The constructor's first argument must be a string + matching one of the [known error + identifiers](https://github.com/ether/etherpad-lite/blob/1.8.16/src/static/js/pad_impexp.js#L80-L86). + +Example: + +```javascript +exports.import = async (hookName, {fileEnding, ImportError}) => { + // Reject all *.etherpad imports with a permission denied message. + if (fileEnding === '.etherpad') throw new ImportError('permission'); +}; +``` ## `userJoin` diff --git a/src/node/handler/ImportHandler.js b/src/node/handler/ImportHandler.js index c865dcf98..9f2a53843 100644 --- a/src/node/handler/ImportHandler.js +++ b/src/node/handler/ImportHandler.js @@ -142,8 +142,8 @@ const doImport = async (req, res, padId) => { } const destFile = path.join(tmpDirectory, `etherpad_import_${randNum}.${exportExtension}`); - const importHandledByPlugin = - (await hooks.aCallAll('import', {srcFile, destFile, fileEnding, padId})).some((x) => x); + const context = {srcFile, destFile, fileEnding, padId, ImportError}; + const importHandledByPlugin = (await hooks.aCallAll('import', context)).some((x) => x); const fileIsEtherpad = (fileEnding === '.etherpad'); const fileIsHTML = (fileEnding === '.html' || fileEnding === '.htm'); const fileIsTXT = (fileEnding === '.txt'); From b7de4faf42102339c891ad475d89d7d21ec3485a Mon Sep 17 00:00:00 2001 From: Richard Hansen Date: Mon, 14 Jun 2021 14:46:04 -0400 Subject: [PATCH 060/446] checkPlugin: Don't bump version if there are no changes --- src/bin/plugins/lib/npmpublish.yml | 21 ++++++++++++++------- 1 file changed, 14 insertions(+), 7 deletions(-) diff --git a/src/bin/plugins/lib/npmpublish.yml b/src/bin/plugins/lib/npmpublish.yml index 4a930144e..f05a532da 100644 --- a/src/bin/plugins/lib/npmpublish.yml +++ b/src/bin/plugins/lib/npmpublish.yml @@ -56,15 +56,22 @@ jobs: runs-on: ubuntu-latest steps: - uses: actions/checkout@v2 + with: + fetch-depth: 0 - uses: actions/setup-node@v1 with: node-version: 12 registry-url: https://registry.npmjs.org/ - - run: git config user.name 'github-actions[bot]' - - run: git config user.email '41898282+github-actions[bot]@users.noreply.github.com' - - run: npm ci - - run: npm version patch - - run: git push --follow-tags + - name: Bump version (patch) + run: | + LATEST_TAG=$(git describe --tags --abbrev=0) || exit 1 + NEW_COMMITS=$(git rev-list --count "${LATEST_TAG}"..) || exit 1 + [ "${NEW_COMMITS}" -gt 0 ] || exit 0 + git config user.name 'github-actions[bot]' + git config user.email '41898282+github-actions[bot]@users.noreply.github.com' + npm ci + npm version patch + git push --follow-tags # `npm publish` must come after `git push` otherwise there is a race # condition: If two PRs are merged back-to-back then master/main will be # updated with the commits from the second PR before the first PR's @@ -79,5 +86,5 @@ jobs: env: NODE_AUTH_TOKEN: ${{secrets.NPM_TOKEN}} -##ETHERPAD_NPM_V=2 -## NPM configuration automatically created using src/bin/plugins/updateAllPluginsScript.sh +# Automatically generated by src/bin/plugins/checkPlugin.js +##ETHERPAD_NPM_V=3 From f0ab112c2d79ee29b23a1b0b7f31ba1acdeefe18 Mon Sep 17 00:00:00 2001 From: Richard Hansen Date: Mon, 14 Jun 2021 19:23:04 -0400 Subject: [PATCH 061/446] checkPlugin: Factor out duplicate file update logic and simplify --- src/bin/plugins/checkPlugin.js | 100 +++++++++------------------------ 1 file changed, 26 insertions(+), 74 deletions(-) diff --git a/src/bin/plugins/checkPlugin.js b/src/bin/plugins/checkPlugin.js index 4876c1362..e7c45f570 100755 --- a/src/bin/plugins/checkPlugin.js +++ b/src/bin/plugins/checkPlugin.js @@ -13,8 +13,10 @@ // unhandled rejection into an uncaught exception, which does cause Node.js to exit. process.on('unhandledRejection', (err) => { throw err; }); +const assert = require('assert').strict; const fs = require('fs'); const childProcess = require('child_process'); +const path = require('path'); // get plugin name & path from user input const pluginName = process.argv[2]; @@ -86,6 +88,28 @@ const prepareRepo = () => { } }; +const checkFile = (srcFn, dstFn) => { + const outFn = path.join(pluginPath, dstFn); + const verRegEx = /^##ETHERPAD_NPM_V=(\d+)$/; + const wantContents = fs.readFileSync(srcFn, {encoding: 'utf8'}); + const [, wantVer] = verRegEx.exec(wantContents) || []; + let gotContents = null; + try { + gotContents = fs.readFileSync(outFn, {encoding: 'utf8'}); + } catch (err) { /* treat as if the file doesn't exist */ } + const [, gotVer] = verRegEx.exec(gotContents || '') || []; + try { + assert.equal(gotVer, wantVer); + } catch (err) { + console.warn(`File ${dstFn} is out of date`); + console.warn(err.message); + if (autoFix) { + fs.mkdirSync(path.dirname(outFn), {recursive: true}); + fs.writeFileSync(outFn, wantContents); + } + } +}; + if (autoCommit) { console.warn('Auto commit is enabled, I hope you know what you are doing...'); } @@ -111,80 +135,8 @@ fs.readdir(pluginPath, (err, rootFiles) => { if (files.indexOf('.git') === -1) throw new Error('No .git folder, aborting'); prepareRepo(); - try { - const path = `${pluginPath}/.github/workflows/npmpublish.yml`; - if (!fs.existsSync(path)) { - console.log('no .github/workflows/npmpublish.yml'); - console.log('create one and set npm secret to auto publish to npm on commit'); - if (autoFix) { - const npmpublish = - fs.readFileSync('src/bin/plugins/lib/npmpublish.yml', {encoding: 'utf8', flag: 'r'}); - fs.mkdirSync(`${pluginPath}/.github/workflows`, {recursive: true}); - fs.writeFileSync(path, npmpublish); - console.log("If you haven't already, setup autopublish for this plugin https://github.com/ether/etherpad-lite/wiki/Plugins:-Automatically-publishing-to-npm-on-commit-to-Github-Repo"); - } else { - console.log('Setup autopublish for this plugin https://github.com/ether/etherpad-lite/wiki/Plugins:-Automatically-publishing-to-npm-on-commit-to-Github-Repo'); - } - } else { - // autopublish exists, we should check the version.. - // checkVersion takes two file paths and checks for a version string in them. - const currVersionFile = fs.readFileSync(path, {encoding: 'utf8', flag: 'r'}); - const existingConfigLocation = currVersionFile.indexOf('##ETHERPAD_NPM_V='); - const existingValue = parseInt( - currVersionFile.substr(existingConfigLocation + 17, existingConfigLocation.length)); - - const reqVersionFile = - fs.readFileSync('src/bin/plugins/lib/npmpublish.yml', {encoding: 'utf8', flag: 'r'}); - const reqConfigLocation = reqVersionFile.indexOf('##ETHERPAD_NPM_V='); - const reqValue = - parseInt(reqVersionFile.substr(reqConfigLocation + 17, reqConfigLocation.length)); - - if (!existingValue || (reqValue > existingValue)) { - const npmpublish = - fs.readFileSync('src/bin/plugins/lib/npmpublish.yml', {encoding: 'utf8', flag: 'r'}); - fs.mkdirSync(`${pluginPath}/.github/workflows`, {recursive: true}); - fs.writeFileSync(path, npmpublish); - } - } - } catch (err) { - console.error(err); - } - - - try { - const path = `${pluginPath}/.github/workflows/backend-tests.yml`; - if (!fs.existsSync(path)) { - console.log('no .github/workflows/backend-tests.yml'); - console.log('create one and set npm secret to auto publish to npm on commit'); - if (autoFix) { - const backendTests = - fs.readFileSync('src/bin/plugins/lib/backend-tests.yml', {encoding: 'utf8', flag: 'r'}); - fs.mkdirSync(`${pluginPath}/.github/workflows`, {recursive: true}); - fs.writeFileSync(path, backendTests); - } - } else { - // autopublish exists, we should check the version.. - // checkVersion takes two file paths and checks for a version string in them. - const currVersionFile = fs.readFileSync(path, {encoding: 'utf8', flag: 'r'}); - const existingConfigLocation = currVersionFile.indexOf('##ETHERPAD_NPM_V='); - const existingValue = parseInt( - currVersionFile.substr(existingConfigLocation + 17, existingConfigLocation.length)); - - const reqVersionFile = - fs.readFileSync('src/bin/plugins/lib/backend-tests.yml', {encoding: 'utf8', flag: 'r'}); - const reqConfigLocation = reqVersionFile.indexOf('##ETHERPAD_NPM_V='); - const reqValue = - parseInt(reqVersionFile.substr(reqConfigLocation + 17, reqConfigLocation.length)); - - if (!existingValue || (reqValue > existingValue)) { - const backendTests = - fs.readFileSync('src/bin/plugins/lib/backend-tests.yml', {encoding: 'utf8', flag: 'r'}); - fs.mkdirSync(`${pluginPath}/.github/workflows`, {recursive: true}); - fs.writeFileSync(path, backendTests); - } - } - } catch (err) { - console.error(err); + for (const fn of ['backend-tests.yml', 'npmpublish.yml']) { + checkFile(`src/bin/plugins/lib/${fn}`, `.github/workflows/${fn}`); } if (files.indexOf('package.json') === -1) { From b7dce95802aaab89f3219a257ab8fca66b6f2569 Mon Sep 17 00:00:00 2001 From: Richard Hansen Date: Mon, 14 Jun 2021 19:27:33 -0400 Subject: [PATCH 062/446] checkPlugin: Use `updateDeps` to manage `engine` --- src/bin/plugins/checkPlugin.js | 15 ++++----------- 1 file changed, 4 insertions(+), 11 deletions(-) diff --git a/src/bin/plugins/checkPlugin.js b/src/bin/plugins/checkPlugin.js index e7c45f570..44c008a6f 100755 --- a/src/bin/plugins/checkPlugin.js +++ b/src/bin/plugins/checkPlugin.js @@ -188,6 +188,10 @@ fs.readdir(pluginPath, (err, rootFiles) => { 'ep_etherpad-lite': {ver: '>=1.8.6', overwrite: false}, }); + updateDeps(parsedPackageJSON, 'engines', { + node: '>=12.13.0', + }); + if (packageJSON.toLowerCase().indexOf('eslintconfig') === -1) { console.warn('No esLintConfig in package.json'); if (autoFix) { @@ -211,17 +215,6 @@ fs.readdir(pluginPath, (err, rootFiles) => { writePackageJson(parsedPackageJSON); } } - - if ((packageJSON.toLowerCase().indexOf('engines') === -1) || !parsedPackageJSON.engines.node) { - console.warn('No engines or node engine in package.json'); - if (autoFix) { - const engines = { - node: '>=12.13.0', - }; - parsedPackageJSON.engines = engines; - writePackageJson(parsedPackageJSON); - } - } } if (files.indexOf('package-lock.json') === -1) { From 314b67b7fe3376218146b9f99458e887008a1b4b Mon Sep 17 00:00:00 2001 From: Richard Hansen Date: Mon, 14 Jun 2021 19:31:26 -0400 Subject: [PATCH 063/446] checkPlugin: Improve `eslintConfig`, `funding`, `scripts` checking --- src/bin/plugins/checkPlugin.js | 69 +++++++++++++++++----------------- 1 file changed, 34 insertions(+), 35 deletions(-) diff --git a/src/bin/plugins/checkPlugin.js b/src/bin/plugins/checkPlugin.js index 44c008a6f..d3b932a82 100755 --- a/src/bin/plugins/checkPlugin.js +++ b/src/bin/plugins/checkPlugin.js @@ -42,6 +42,23 @@ const writePackageJson = (obj) => { return fs.writeFileSync(`${pluginPath}/package.json`, s); }; +const checkEntries = (got, want) => { + let changed = false; + for (const [key, val] of Object.entries(want)) { + try { + assert.deepEqual(got[key], val); + } catch (err) { + console.warn(`${key} possibly outdated.`); + console.warn(err.message); + if (autoFix) { + got[key] = val; + changed = true; + } + } + } + return changed; +}; + const updateDeps = (parsedPackageJson, key, wantDeps) => { const {[key]: deps = {}} = parsedPackageJson; let changed = false; @@ -147,19 +164,6 @@ fs.readdir(pluginPath, (err, rootFiles) => { const packageJSON = fs.readFileSync(`${pluginPath}/package.json`, {encoding: 'utf8', flag: 'r'}); const parsedPackageJSON = JSON.parse(packageJSON); - if (autoFix) { - let updatedPackageJSON = false; - if (!parsedPackageJSON.funding) { - updatedPackageJSON = true; - parsedPackageJSON.funding = { - type: 'individual', - url: 'https://etherpad.org/', - }; - } - if (updatedPackageJSON) { - writePackageJson(parsedPackageJSON); - } - } if (packageJSON.toLowerCase().indexOf('repository') === -1) { console.warn('No repository in package.json'); @@ -192,29 +196,24 @@ fs.readdir(pluginPath, (err, rootFiles) => { node: '>=12.13.0', }); - if (packageJSON.toLowerCase().indexOf('eslintconfig') === -1) { - console.warn('No esLintConfig in package.json'); - if (autoFix) { - const eslintConfig = { - root: true, - extends: 'etherpad/plugin', - }; - parsedPackageJSON.eslintConfig = eslintConfig; - writePackageJson(parsedPackageJSON); - } - } + if (parsedPackageJSON.eslintConfig == null) parsedPackageJSON.eslintConfig = {}; + if (checkEntries(parsedPackageJSON.eslintConfig, { + root: true, + extends: 'etherpad/plugin', + })) await writePackageJson(parsedPackageJSON); - if (packageJSON.toLowerCase().indexOf('scripts') === -1) { - console.warn('No scripts in package.json'); - if (autoFix) { - const scripts = { - 'lint': 'eslint .', - 'lint:fix': 'eslint --fix .', - }; - parsedPackageJSON.scripts = scripts; - writePackageJson(parsedPackageJSON); - } - } + if (checkEntries(parsedPackageJSON, { + funding: { + type: 'individual', + url: 'https://etherpad.org/', + }, + })) writePackageJson(parsedPackageJSON); + + if (parsedPackageJSON.scripts == null) parsedPackageJSON.scripts = {}; + if (checkEntries(parsedPackageJSON.scripts, { + 'lint': 'eslint .', + 'lint:fix': 'eslint --fix .', + })) writePackageJson(parsedPackageJSON); } if (files.indexOf('package-lock.json') === -1) { From f0669a8d31bbffbf8ca687dadcd3c3fa5e9f6ad3 Mon Sep 17 00:00:00 2001 From: Richard Hansen Date: Mon, 14 Jun 2021 19:36:20 -0400 Subject: [PATCH 064/446] checkPlugin: Automatically determine plugin name in `backend-tests.yml` --- src/bin/plugins/lib/backend-tests.yml | 30 ++++++++++++++++++++------- 1 file changed, 22 insertions(+), 8 deletions(-) diff --git a/src/bin/plugins/lib/backend-tests.yml b/src/bin/plugins/lib/backend-tests.yml index 3cb7dad50..a4a01530d 100644 --- a/src/bin/plugins/lib/backend-tests.yml +++ b/src/bin/plugins/lib/backend-tests.yml @@ -1,6 +1,3 @@ -# You need to change lines 38 and 46 in case the plugin's name on npmjs.com is different -# from the repository name - name: "Backend tests" # any branch is useful for testing before a PR is submitted @@ -17,6 +14,10 @@ jobs: runs-on: ubuntu-latest steps: + - uses: actions/setup-node@v2 + with: + node-version: 12 + - name: Install libreoffice run: | sudo add-apt-repository -y ppa:libreoffice/ppa @@ -32,19 +33,32 @@ jobs: - name: Install all dependencies and symlink for ep_etherpad-lite run: src/bin/installDeps.sh - # clone this repository into node_modules/ep_plugin-name - name: Checkout plugin repository uses: actions/checkout@v2 with: - path: ./node_modules/${{github.event.repository.name}} + path: ./node_modules/__tmp + + - name: Determine plugin name + id: plugin_name + run: | + cd ./node_modules/__tmp + npx -c 'printf %s\\n "::set-output name=plugin_name::${npm_package_name}"' + + - name: Rename plugin directory + run: | + mv ./node_modules/__tmp ./node_modules/"${PLUGIN_NAME}" + env: + PLUGIN_NAME: ${{ steps.plugin_name.outputs.plugin_name }} - name: Install plugin dependencies run: | - cd node_modules/${{github.event.repository.name}} + cd ./node_modules/"${PLUGIN_NAME}" npm ci + env: + PLUGIN_NAME: ${{ steps.plugin_name.outputs.plugin_name }} - name: Run the backend tests run: cd src && npm test -##ETHERPAD_NPM_V=2 -## NPM configuration automatically created using src/bin/plugins/updateAllPluginsScript.sh +# Automatically generated by src/bin/plugins/checkPlugin.js +##ETHERPAD_NPM_V=3 From 51c530a3a0798888daf8b6e7fcabe7f512f56024 Mon Sep 17 00:00:00 2001 From: Richard Hansen Date: Mon, 14 Jun 2021 19:54:37 -0400 Subject: [PATCH 065/446] checkPlugin: Compare entire file --- src/bin/plugins/checkPlugin.js | 5 +---- src/bin/plugins/lib/backend-tests.yml | 3 --- src/bin/plugins/lib/npmpublish.yml | 3 --- 3 files changed, 1 insertion(+), 10 deletions(-) diff --git a/src/bin/plugins/checkPlugin.js b/src/bin/plugins/checkPlugin.js index d3b932a82..ed94365d1 100755 --- a/src/bin/plugins/checkPlugin.js +++ b/src/bin/plugins/checkPlugin.js @@ -107,16 +107,13 @@ const prepareRepo = () => { const checkFile = (srcFn, dstFn) => { const outFn = path.join(pluginPath, dstFn); - const verRegEx = /^##ETHERPAD_NPM_V=(\d+)$/; const wantContents = fs.readFileSync(srcFn, {encoding: 'utf8'}); - const [, wantVer] = verRegEx.exec(wantContents) || []; let gotContents = null; try { gotContents = fs.readFileSync(outFn, {encoding: 'utf8'}); } catch (err) { /* treat as if the file doesn't exist */ } - const [, gotVer] = verRegEx.exec(gotContents || '') || []; try { - assert.equal(gotVer, wantVer); + assert.equal(gotContents, wantContents); } catch (err) { console.warn(`File ${dstFn} is out of date`); console.warn(err.message); diff --git a/src/bin/plugins/lib/backend-tests.yml b/src/bin/plugins/lib/backend-tests.yml index a4a01530d..0451596f8 100644 --- a/src/bin/plugins/lib/backend-tests.yml +++ b/src/bin/plugins/lib/backend-tests.yml @@ -59,6 +59,3 @@ jobs: - name: Run the backend tests run: cd src && npm test - -# Automatically generated by src/bin/plugins/checkPlugin.js -##ETHERPAD_NPM_V=3 diff --git a/src/bin/plugins/lib/npmpublish.yml b/src/bin/plugins/lib/npmpublish.yml index f05a532da..5bbc1ca8c 100644 --- a/src/bin/plugins/lib/npmpublish.yml +++ b/src/bin/plugins/lib/npmpublish.yml @@ -85,6 +85,3 @@ jobs: - run: npm publish env: NODE_AUTH_TOKEN: ${{secrets.NPM_TOKEN}} - -# Automatically generated by src/bin/plugins/checkPlugin.js -##ETHERPAD_NPM_V=3 From 48222449b5739d3abdcc4ccef11583274aa31194 Mon Sep 17 00:00:00 2001 From: Richard Hansen Date: Mon, 14 Jun 2021 19:55:18 -0400 Subject: [PATCH 066/446] checkPlugin: Add `frontend-tests.yml` GitHub workflow --- src/bin/plugins/checkPlugin.js | 2 +- src/bin/plugins/lib/frontend-tests.yml | 92 ++++++++++++++++++++++++++ 2 files changed, 93 insertions(+), 1 deletion(-) create mode 100644 src/bin/plugins/lib/frontend-tests.yml diff --git a/src/bin/plugins/checkPlugin.js b/src/bin/plugins/checkPlugin.js index ed94365d1..843c22964 100755 --- a/src/bin/plugins/checkPlugin.js +++ b/src/bin/plugins/checkPlugin.js @@ -149,7 +149,7 @@ fs.readdir(pluginPath, (err, rootFiles) => { if (files.indexOf('.git') === -1) throw new Error('No .git folder, aborting'); prepareRepo(); - for (const fn of ['backend-tests.yml', 'npmpublish.yml']) { + for (const fn of ['backend-tests.yml', 'frontend-tests.yml', 'npmpublish.yml']) { checkFile(`src/bin/plugins/lib/${fn}`, `.github/workflows/${fn}`); } diff --git a/src/bin/plugins/lib/frontend-tests.yml b/src/bin/plugins/lib/frontend-tests.yml new file mode 100644 index 000000000..547b23716 --- /dev/null +++ b/src/bin/plugins/lib/frontend-tests.yml @@ -0,0 +1,92 @@ +# Publicly credit Sauce Labs because they generously support open source +# projects. +name: "frontend tests powered by Sauce Labs" + +on: [push] + +jobs: + test: + runs-on: ubuntu-latest + + steps: + - name: Generate Sauce Labs strings + id: sauce_strings + run: | + printf %s\\n '::set-output name=name::${{github.event.repository.name}} ${{ github.workflow }} - ${{ github.job }}' + printf %s\\n '::set-output name=tunnel_id::${{ github.run_id }}-${{ github.run_number }}-${{ github.job }}' + + - uses: actions/setup-node@v2 + with: + node-version: 12 + + - name: Check out Etherpad core + uses: actions/checkout@v2 + with: + repository: ether/etherpad-lite + + - name: Check out the plugin + uses: actions/checkout@v2 + with: + path: ./node_modules/__tmp + + - name: export GIT_HASH to env + id: environment + run: | + cd ./node_modules/__tmp + echo "::set-output name=sha_short::$(git rev-parse --short ${{ github.sha }})" + + - name: Determine plugin name + id: plugin_name + run: | + cd ./node_modules/__tmp + npx -c 'printf %s\\n "::set-output name=plugin_name::${npm_package_name}"' + + - name: Rename plugin directory + env: + PLUGIN_NAME: ${{ steps.plugin_name.outputs.plugin_name }} + run: | + mv ./node_modules/__tmp ./node_modules/"${PLUGIN_NAME}" + + - name: Install plugin dependencies + env: + PLUGIN_NAME: ${{ steps.plugin_name.outputs.plugin_name }} + run: | + cd ./node_modules/"${PLUGIN_NAME}" + npm ci + + # This must be run after setting up the plugin, otherwise npm will try to + # hoist common dependencies by removing them from src/node_modules and + # installing them in the top-level node_modules. As of v6.14.10, npm's hoist + # logic appears to be buggy, because it sometimes removes dependencies from + # src/node_modules but fails to add them to the top-level node_modules. Even + # if npm correctly hoists the dependencies, the hoisting seems to confuse + # tools such as `npm outdated`, `npm update`, and some ESLint rules. + - name: Install Etherpad core dependencies + run: src/bin/installDeps.sh + + - name: Create settings.json + run: cp settings.json.template settings.json + + - name: Disable import/export rate limiting + run: | + sed -e '/^ *"importExportRateLimiting":/,/^ *\}/ s/"max":.*/"max": 0/' -i settings.json + + - name: Remove standard frontend test files + run: rm -rf src/tests/frontend/specs + + - uses: saucelabs/sauce-connect-action@v1 + with: + username: ${{ secrets.SAUCE_USERNAME }} + accessKey: ${{ secrets.SAUCE_ACCESS_KEY }} + tunnelIdentifier: ${{ steps.sauce_strings.outputs.tunnel_id }} + + - name: Run the frontend tests + shell: bash + env: + SAUCE_USERNAME: ${{ secrets.SAUCE_USERNAME }} + SAUCE_ACCESS_KEY: ${{ secrets.SAUCE_ACCESS_KEY }} + SAUCE_NAME: ${{ steps.sauce_strings.outputs.name }} + TRAVIS_JOB_NUMBER: ${{ steps.sauce_strings.outputs.tunnel_id }} + GIT_HASH: ${{ steps.environment.outputs.sha_short }} + run: | + src/tests/frontend/travis/runner.sh From 34a4a74634481db11634374a93c0218b7410bca6 Mon Sep 17 00:00:00 2001 From: Richard Hansen Date: Wed, 16 Jun 2021 18:41:26 -0400 Subject: [PATCH 067/446] checkPlugin: Change `autocommit` to not push --- src/bin/plugins/README.md | 8 ++++++- src/bin/plugins/checkPlugin.js | 29 +++++++++++++++-------- src/bin/plugins/reTestAllPlugins.sh | 2 +- src/bin/plugins/updateAllPluginsScript.sh | 2 +- src/bin/plugins/updateCorePlugins.sh | 2 +- 5 files changed, 29 insertions(+), 14 deletions(-) diff --git a/src/bin/plugins/README.md b/src/bin/plugins/README.md index b14065821..10602f987 100755 --- a/src/bin/plugins/README.md +++ b/src/bin/plugins/README.md @@ -21,12 +21,18 @@ node src/bin/plugins/checkPlugin.js ep_webrtc node src/bin/plugins/checkPlugin.js ep_whatever autofix ``` -## Autocommitting, push, npm minor patch and npm publish (highly dangerous) +## Autocommitting - fix issues and commit ``` node src/bin/plugins/checkPlugin.js ep_whatever autocommit ``` +## Autopush - fix issues, commit, push, and publish (highly dangerous) + +``` +node src/bin/plugins/checkPlugin.js ep_whatever autopush +``` + # All the plugins Replace johnmclear with your github username diff --git a/src/bin/plugins/checkPlugin.js b/src/bin/plugins/checkPlugin.js index 843c22964..697e339bc 100755 --- a/src/bin/plugins/checkPlugin.js +++ b/src/bin/plugins/checkPlugin.js @@ -5,8 +5,9 @@ * * Normal usage: node src/bin/plugins/checkPlugin.js ep_whatever * Auto fix the things it can: node src/bin/plugins/checkPlugin.js ep_whatever autofix - * Auto commit, push and publish to npm (highly dangerous): - * node src/bin/plugins/checkPlugin.js ep_whatever autocommit + * Auto fix and commit: node src/bin/plugins/checkPlugin.js ep_whatever autocommit + * Auto fix, commit, push and publish to npm (highly dangerous): + * node src/bin/plugins/checkPlugin.js ep_whatever autopush */ // As of v14, Node.js does not exit when there is an unhandled Promise rejection. Convert an @@ -28,7 +29,8 @@ const pluginPath = `node_modules/${pluginName}`; console.log(`Checking the plugin: ${pluginName}`); const optArgs = process.argv.slice(3); -const autoCommit = optArgs.indexOf('autocommit') !== -1; +const autoPush = optArgs.indexOf('autopush') !== -1; +const autoCommit = autoPush || optArgs.indexOf('autocommit') !== -1; const autoFix = autoCommit || optArgs.indexOf('autofix') !== -1; const execSync = (cmd, opts = {}) => (childProcess.execSync(cmd, { @@ -124,8 +126,8 @@ const checkFile = (srcFn, dstFn) => { } }; -if (autoCommit) { - console.warn('Auto commit is enabled, I hope you know what you are doing...'); +if (autoPush) { + console.warn('Auto push is enabled, I hope you know what you are doing...'); } fs.readdir(pluginPath, (err, rootFiles) => { @@ -379,17 +381,24 @@ fs.readdir(pluginPath, (err, rootFiles) => { }); fs.unlinkSync(`${pluginPath}/.git/checkPlugin.index`); - const cmd = [ + const commitCmd = [ 'git add -A', 'git commit -m "autofixes from Etherpad checkPlugin.js"', - 'git push', ].join(' && '); if (autoCommit) { - console.log('Attempting autocommit and auto publish to npm'); - execSync(cmd, {stdio: 'inherit'}); + console.log('Committing changes...'); + execSync(commitCmd, {stdio: 'inherit'}); } else { console.log('Fixes applied. Check the above git diff then run the following command:'); - console.log(`(cd node_modules/${pluginName} && ${cmd})`); + console.log(`(cd node_modules/${pluginName} && ${commitCmd})`); + } + const pushCmd = 'git push'; + if (autoPush) { + console.log('Pushing new commit...'); + execSync(pushCmd, {stdio: 'inherit'}); + } else { + console.log('Changes committed. To push, run the following command:'); + console.log(`(cd node_modules/${pluginName} && ${pushCmd})`); } } else { console.log('No changes.'); diff --git a/src/bin/plugins/reTestAllPlugins.sh b/src/bin/plugins/reTestAllPlugins.sh index 58628bdb0..abe1bca80 100755 --- a/src/bin/plugins/reTestAllPlugins.sh +++ b/src/bin/plugins/reTestAllPlugins.sh @@ -4,7 +4,7 @@ do echo $dir if [[ $dir == *"ep_"* ]]; then if [[ $dir != "ep_etherpad-lite" ]]; then - # node src/bin/plugins/checkPlugin.js $dir autofix autocommit autoupdate + # node src/bin/plugins/checkPlugin.js $dir autopush cd node_modules/$dir git commit -m "Automatic update: bump update to re-run latest Etherpad tests" --allow-empty git push origin master diff --git a/src/bin/plugins/updateAllPluginsScript.sh b/src/bin/plugins/updateAllPluginsScript.sh index bf5280ee0..79be4bc47 100755 --- a/src/bin/plugins/updateAllPluginsScript.sh +++ b/src/bin/plugins/updateAllPluginsScript.sh @@ -10,7 +10,7 @@ do # echo $0 if [[ $dir == *"ep_"* ]]; then if [[ $dir != "ep_etherpad-lite" ]]; then - node src/bin/plugins/checkPlugin.js $dir autofix autocommit autoupdate + node src/bin/plugins/checkPlugin.js $dir autopush fi fi # echo $dir diff --git a/src/bin/plugins/updateCorePlugins.sh b/src/bin/plugins/updateCorePlugins.sh index 402a080ec..3866b8444 100755 --- a/src/bin/plugins/updateCorePlugins.sh +++ b/src/bin/plugins/updateCorePlugins.sh @@ -5,5 +5,5 @@ set -e for dir in node_modules/ep_*; do dir=${dir#node_modules/} [ "$dir" != ep_etherpad-lite ] || continue - node src/bin/plugins/checkPlugin.js "$dir" autofix autocommit autoupdate + node src/bin/plugins/checkPlugin.js "$dir" autopush done From b546867adb0eaf3377b92fe5603f8d7af88d2546 Mon Sep 17 00:00:00 2001 From: Richard Hansen Date: Thu, 17 Jun 2021 16:50:23 -0400 Subject: [PATCH 068/446] checkPlugin: Replace `.indexOf()` with `.includes()` --- src/bin/plugins/checkPlugin.js | 46 ++++++++++++++++------------------ 1 file changed, 22 insertions(+), 24 deletions(-) diff --git a/src/bin/plugins/checkPlugin.js b/src/bin/plugins/checkPlugin.js index 697e339bc..8d983c62f 100755 --- a/src/bin/plugins/checkPlugin.js +++ b/src/bin/plugins/checkPlugin.js @@ -29,9 +29,9 @@ const pluginPath = `node_modules/${pluginName}`; console.log(`Checking the plugin: ${pluginName}`); const optArgs = process.argv.slice(3); -const autoPush = optArgs.indexOf('autopush') !== -1; -const autoCommit = autoPush || optArgs.indexOf('autocommit') !== -1; -const autoFix = autoCommit || optArgs.indexOf('autofix') !== -1; +const autoPush = optArgs.includes('autopush'); +const autoCommit = autoPush || optArgs.includes('autocommit'); +const autoFix = autoCommit || optArgs.includes('autofix'); const execSync = (cmd, opts = {}) => (childProcess.execSync(cmd, { cwd: `${pluginPath}/`, @@ -144,27 +144,25 @@ fs.readdir(pluginPath, (err, rootFiles) => { let repository; for (let i = 0; i < rootFiles.length; i++) { - if (rootFiles[i].toLowerCase().indexOf('readme') !== -1) readMeFileName = rootFiles[i]; + if (rootFiles[i].toLowerCase().includes('readme')) readMeFileName = rootFiles[i]; files.push(rootFiles[i].toLowerCase()); } - if (files.indexOf('.git') === -1) throw new Error('No .git folder, aborting'); + if (!files.includes('.git')) throw new Error('No .git folder, aborting'); prepareRepo(); for (const fn of ['backend-tests.yml', 'frontend-tests.yml', 'npmpublish.yml']) { checkFile(`src/bin/plugins/lib/${fn}`, `.github/workflows/${fn}`); } - if (files.indexOf('package.json') === -1) { + if (!files.includes('package.json')) { console.warn('no package.json, please create'); - } - - if (files.indexOf('package.json') !== -1) { + } else { const packageJSON = fs.readFileSync(`${pluginPath}/package.json`, {encoding: 'utf8', flag: 'r'}); const parsedPackageJSON = JSON.parse(packageJSON); - if (packageJSON.toLowerCase().indexOf('repository') === -1) { + if (!packageJSON.toLowerCase().includes('repository')) { console.warn('No repository in package.json'); if (autoFix) { console.warn('Repository not detected in package.json. Add repository section.'); @@ -215,13 +213,13 @@ fs.readdir(pluginPath, (err, rootFiles) => { })) writePackageJson(parsedPackageJSON); } - if (files.indexOf('package-lock.json') === -1) { + if (!files.includes('package-lock.json')) { console.warn('package-lock.json not found'); if (!autoFix) { console.warn('Run npm install in the plugin folder and commit the package-lock.json file.'); } } - if (files.indexOf('readme') === -1 && files.indexOf('readme.md') === -1) { + if (!files.includes('readme') && !files.includes('readme.md')) { console.warn('README.md file not found, please create'); if (autoFix) { console.log('Autofixing missing README.md file'); @@ -240,7 +238,7 @@ fs.readdir(pluginPath, (err, rootFiles) => { } } - if (files.indexOf('contributing') === -1 && files.indexOf('contributing.md') === -1) { + if (!files.includes('contributing') && !files.includes('contributing.md')) { console.warn('CONTRIBUTING.md file not found, please create'); if (autoFix) { console.log('Autofixing missing CONTRIBUTING.md file, please edit the CONTRIBUTING.md ' + @@ -256,7 +254,7 @@ fs.readdir(pluginPath, (err, rootFiles) => { if (readMeFileName) { let readme = fs.readFileSync(`${pluginPath}/${readMeFileName}`, {encoding: 'utf8', flag: 'r'}); - if (readme.toLowerCase().indexOf('license') === -1) { + if (!readme.toLowerCase().includes('license')) { console.warn('No license section in README'); if (autoFix) { console.warn('Please add License section to README manually.'); @@ -266,10 +264,10 @@ fs.readdir(pluginPath, (err, rootFiles) => { const publishBadge = `![Publish Status](https://github.com/ether/${pluginName}/workflows/Node.js%20Package/badge.svg)`; // eslint-disable-next-line max-len const testBadge = `![Backend Tests Status](https://github.com/ether/${pluginName}/workflows/Backend%20tests/badge.svg)`; - if (readme.toLowerCase().indexOf('travis') !== -1) { + if (readme.toLowerCase().includes('travis')) { console.warn('Remove Travis badges'); } - if (readme.indexOf('workflows/Node.js%20Package/badge.svg') === -1) { + if (!readme.includes('workflows/Node.js%20Package/badge.svg')) { console.warn('No Github workflow badge detected'); if (autoFix) { readme = `${publishBadge} ${testBadge}\n\n${readme}`; @@ -280,7 +278,7 @@ fs.readdir(pluginPath, (err, rootFiles) => { } } - if (files.indexOf('license') === -1 && files.indexOf('license.md') === -1) { + if (!files.includes('license') && !files.includes('license.md')) { console.warn('LICENSE.md file not found, please create'); if (autoFix) { console.log('Autofixing missing LICENSE.md file, including Apache 2 license.'); @@ -292,7 +290,7 @@ fs.readdir(pluginPath, (err, rootFiles) => { } } - if (files.indexOf('.gitignore') === -1) { + if (!files.includes('.gitignore')) { console.warn('.gitignore file not found, please create. .gitignore files are useful to ' + "ensure files aren't incorrectly commited to a repository."); if (autoFix) { @@ -304,7 +302,7 @@ fs.readdir(pluginPath, (err, rootFiles) => { } else { let gitignore = fs.readFileSync(`${pluginPath}/.gitignore`, {encoding: 'utf8', flag: 'r'}); - if (gitignore.indexOf('node_modules/') === -1) { + if (!gitignore.includes('node_modules/')) { console.warn('node_modules/ missing from .gitignore'); if (autoFix) { gitignore += 'node_modules/'; @@ -314,13 +312,13 @@ fs.readdir(pluginPath, (err, rootFiles) => { } // if we include templates but don't have translations... - if (files.indexOf('templates') !== -1 && files.indexOf('locales') === -1) { + if (files.includes('templates') && !files.includes('locales')) { console.warn('Translations not found, please create. ' + 'Translation files help with Etherpad accessibility.'); } - if (files.indexOf('.ep_initialized') !== -1) { + if (files.includes('.ep_initialized')) { console.warn( '.ep_initialized found, please remove. .ep_initialized should never be commited to git ' + 'and should only exist once the plugin has been executed one time.'); @@ -330,7 +328,7 @@ fs.readdir(pluginPath, (err, rootFiles) => { } } - if (files.indexOf('npm-debug.log') !== -1) { + if (files.includes('npm-debug.log')) { console.warn('npm-debug.log found, please remove. npm-debug.log should never be commited to ' + 'your repository.'); if (autoFix) { @@ -339,9 +337,9 @@ fs.readdir(pluginPath, (err, rootFiles) => { } } - if (files.indexOf('static') !== -1) { + if (files.includes('static')) { fs.readdir(`${pluginPath}/static`, (errRead, staticFiles) => { - if (staticFiles.indexOf('tests') === -1) { + if (!staticFiles.includes('tests')) { console.warn('Test files not found, please create tests. https://github.com/ether/etherpad-lite/wiki/Creating-a-plugin#writing-and-running-front-end-tests-for-your-plugin'); } }); From b50c6d07d4c63c5c78b2eac83eba07a5da1619ac Mon Sep 17 00:00:00 2001 From: Richard Hansen Date: Thu, 17 Jun 2021 17:13:17 -0400 Subject: [PATCH 069/446] checkPlugin: Improve readability of `files` assignment --- src/bin/plugins/checkPlugin.js | 13 +++++-------- 1 file changed, 5 insertions(+), 8 deletions(-) diff --git a/src/bin/plugins/checkPlugin.js b/src/bin/plugins/checkPlugin.js index 8d983c62f..57116737b 100755 --- a/src/bin/plugins/checkPlugin.js +++ b/src/bin/plugins/checkPlugin.js @@ -136,17 +136,14 @@ fs.readdir(pluginPath, (err, rootFiles) => { return console.log(`Unable to scan directory: ${err}`); } - // rewriting files to lower case - const files = []; - // some files we need to know the actual file name. Not compulsory but might help in the future. let readMeFileName; let repository; - - for (let i = 0; i < rootFiles.length; i++) { - if (rootFiles[i].toLowerCase().includes('readme')) readMeFileName = rootFiles[i]; - files.push(rootFiles[i].toLowerCase()); - } + const files = rootFiles.map((f) => { + const fl = f.toLowerCase(); + if (fl.includes('readme')) readMeFileName = f; + return fl; + }); if (!files.includes('.git')) throw new Error('No .git folder, aborting'); prepareRepo(); From 753d16af8a313d1a0e29fff306e046925496eada Mon Sep 17 00:00:00 2001 From: Richard Hansen Date: Thu, 17 Jun 2021 17:19:56 -0400 Subject: [PATCH 070/446] checkPlugin: Promisify file system accesses --- src/bin/plugins/checkPlugin.js | 87 +++++++++++++++++----------------- 1 file changed, 43 insertions(+), 44 deletions(-) diff --git a/src/bin/plugins/checkPlugin.js b/src/bin/plugins/checkPlugin.js index 57116737b..b7a45f3fb 100755 --- a/src/bin/plugins/checkPlugin.js +++ b/src/bin/plugins/checkPlugin.js @@ -16,6 +16,7 @@ process.on('unhandledRejection', (err) => { throw err; }); const assert = require('assert').strict; const fs = require('fs'); +const fsp = fs.promises; const childProcess = require('child_process'); const path = require('path'); @@ -38,10 +39,10 @@ const execSync = (cmd, opts = {}) => (childProcess.execSync(cmd, { ...opts, }) || '').toString().replace(/\n+$/, ''); -const writePackageJson = (obj) => { +const writePackageJson = async (obj) => { let s = JSON.stringify(obj, null, 2); if (s.length && s.slice(s.length - 1) !== '\n') s += '\n'; - return fs.writeFileSync(`${pluginPath}/package.json`, s); + return await fsp.writeFile(`${pluginPath}/package.json`, s); }; const checkEntries = (got, want) => { @@ -61,7 +62,7 @@ const checkEntries = (got, want) => { return changed; }; -const updateDeps = (parsedPackageJson, key, wantDeps) => { +const updateDeps = async (parsedPackageJson, key, wantDeps) => { const {[key]: deps = {}} = parsedPackageJson; let changed = false; for (const [pkg, verInfo] of Object.entries(wantDeps)) { @@ -80,7 +81,7 @@ const updateDeps = (parsedPackageJson, key, wantDeps) => { } if (changed) { parsedPackageJson[key] = deps; - writePackageJson(parsedPackageJson); + await writePackageJson(parsedPackageJson); } }; @@ -107,12 +108,12 @@ const prepareRepo = () => { } }; -const checkFile = (srcFn, dstFn) => { +const checkFile = async (srcFn, dstFn) => { const outFn = path.join(pluginPath, dstFn); - const wantContents = fs.readFileSync(srcFn, {encoding: 'utf8'}); + const wantContents = await fsp.readFile(srcFn, {encoding: 'utf8'}); let gotContents = null; try { - gotContents = fs.readFileSync(outFn, {encoding: 'utf8'}); + gotContents = await fsp.readFile(outFn, {encoding: 'utf8'}); } catch (err) { /* treat as if the file doesn't exist */ } try { assert.equal(gotContents, wantContents); @@ -120,8 +121,8 @@ const checkFile = (srcFn, dstFn) => { console.warn(`File ${dstFn} is out of date`); console.warn(err.message); if (autoFix) { - fs.mkdirSync(path.dirname(outFn), {recursive: true}); - fs.writeFileSync(outFn, wantContents); + await fsp.mkdir(path.dirname(outFn), {recursive: true}); + await fsp.writeFile(outFn, wantContents); } } }; @@ -130,11 +131,8 @@ if (autoPush) { console.warn('Auto push is enabled, I hope you know what you are doing...'); } -fs.readdir(pluginPath, (err, rootFiles) => { - // handling error - if (err) { - return console.log(`Unable to scan directory: ${err}`); - } +(async () => { + const rootFiles = await fsp.readdir(pluginPath); // some files we need to know the actual file name. Not compulsory but might help in the future. let readMeFileName; @@ -148,15 +146,16 @@ fs.readdir(pluginPath, (err, rootFiles) => { if (!files.includes('.git')) throw new Error('No .git folder, aborting'); prepareRepo(); - for (const fn of ['backend-tests.yml', 'frontend-tests.yml', 'npmpublish.yml']) { - checkFile(`src/bin/plugins/lib/${fn}`, `.github/workflows/${fn}`); - } + const workflows = ['backend-tests.yml', 'frontend-tests.yml', 'npmpublish.yml']; + await Promise.all(workflows.map(async (fn) => { + await checkFile(`src/bin/plugins/lib/${fn}`, `.github/workflows/${fn}`); + })); if (!files.includes('package.json')) { console.warn('no package.json, please create'); } else { const packageJSON = - fs.readFileSync(`${pluginPath}/package.json`, {encoding: 'utf8', flag: 'r'}); + await fsp.readFile(`${pluginPath}/package.json`, {encoding: 'utf8', flag: 'r'}); const parsedPackageJSON = JSON.parse(packageJSON); if (!packageJSON.toLowerCase().includes('repository')) { @@ -169,7 +168,7 @@ fs.readdir(pluginPath, (err, rootFiles) => { repository = parsedPackageJSON.repository.url; } - updateDeps(parsedPackageJSON, 'devDependencies', { + await updateDeps(parsedPackageJSON, 'devDependencies', { 'eslint': '^7.28.0', 'eslint-config-etherpad': '^2.0.0', 'eslint-plugin-cypress': '^2.11.3', @@ -181,12 +180,12 @@ fs.readdir(pluginPath, (err, rootFiles) => { 'eslint-plugin-you-dont-need-lodash-underscore': '^6.12.0', }); - updateDeps(parsedPackageJSON, 'peerDependencies', { + await updateDeps(parsedPackageJSON, 'peerDependencies', { // Some plugins require a newer version of Etherpad so don't overwrite if already set. 'ep_etherpad-lite': {ver: '>=1.8.6', overwrite: false}, }); - updateDeps(parsedPackageJSON, 'engines', { + await updateDeps(parsedPackageJSON, 'engines', { node: '>=12.13.0', }); @@ -201,13 +200,13 @@ fs.readdir(pluginPath, (err, rootFiles) => { type: 'individual', url: 'https://etherpad.org/', }, - })) writePackageJson(parsedPackageJSON); + })) await writePackageJson(parsedPackageJSON); if (parsedPackageJSON.scripts == null) parsedPackageJSON.scripts = {}; if (checkEntries(parsedPackageJSON.scripts, { 'lint': 'eslint .', 'lint:fix': 'eslint --fix .', - })) writePackageJson(parsedPackageJSON); + })) await writePackageJson(parsedPackageJSON); } if (!files.includes('package-lock.json')) { @@ -221,14 +220,15 @@ fs.readdir(pluginPath, (err, rootFiles) => { if (autoFix) { console.log('Autofixing missing README.md file'); console.log('please edit the README.md file further to include plugin specific details.'); - let readme = fs.readFileSync('src/bin/plugins/lib/README.md', {encoding: 'utf8', flag: 'r'}); + let readme = + await fsp.readFile('src/bin/plugins/lib/README.md', {encoding: 'utf8', flag: 'r'}); readme = readme.replace(/\[plugin_name\]/g, pluginName); if (repository) { const org = repository.split('/')[3]; const name = repository.split('/')[4]; readme = readme.replace(/\[org_name\]/g, org); readme = readme.replace(/\[repo_url\]/g, name); - fs.writeFileSync(`${pluginPath}/README.md`, readme); + await fsp.writeFile(`${pluginPath}/README.md`, readme); } else { console.warn('Unable to find repository in package.json, aborting.'); } @@ -241,16 +241,16 @@ fs.readdir(pluginPath, (err, rootFiles) => { console.log('Autofixing missing CONTRIBUTING.md file, please edit the CONTRIBUTING.md ' + 'file further to include plugin specific details.'); let contributing = - fs.readFileSync('src/bin/plugins/lib/CONTRIBUTING.md', {encoding: 'utf8', flag: 'r'}); + await fsp.readFile('src/bin/plugins/lib/CONTRIBUTING.md', {encoding: 'utf8', flag: 'r'}); contributing = contributing.replace(/\[plugin_name\]/g, pluginName); - fs.writeFileSync(`${pluginPath}/CONTRIBUTING.md`, contributing); + await fsp.writeFile(`${pluginPath}/CONTRIBUTING.md`, contributing); } } if (readMeFileName) { let readme = - fs.readFileSync(`${pluginPath}/${readMeFileName}`, {encoding: 'utf8', flag: 'r'}); + await fsp.readFile(`${pluginPath}/${readMeFileName}`, {encoding: 'utf8', flag: 'r'}); if (!readme.toLowerCase().includes('license')) { console.warn('No license section in README'); if (autoFix) { @@ -269,7 +269,7 @@ fs.readdir(pluginPath, (err, rootFiles) => { if (autoFix) { readme = `${publishBadge} ${testBadge}\n\n${readme}`; // write readme to file system - fs.writeFileSync(`${pluginPath}/${readMeFileName}`, readme); + await fsp.writeFile(`${pluginPath}/${readMeFileName}`, readme); console.log('Wrote Github workflow badges to README'); } } @@ -280,10 +280,10 @@ fs.readdir(pluginPath, (err, rootFiles) => { if (autoFix) { console.log('Autofixing missing LICENSE.md file, including Apache 2 license.'); let license = - fs.readFileSync('src/bin/plugins/lib/LICENSE.md', {encoding: 'utf8', flag: 'r'}); + await fsp.readFile('src/bin/plugins/lib/LICENSE.md', {encoding: 'utf8', flag: 'r'}); license = license.replace('[yyyy]', new Date().getFullYear()); license = license.replace('[name of copyright owner]', execSync('git config user.name')); - fs.writeFileSync(`${pluginPath}/LICENSE.md`, license); + await fsp.writeFile(`${pluginPath}/LICENSE.md`, license); } } @@ -293,17 +293,17 @@ fs.readdir(pluginPath, (err, rootFiles) => { if (autoFix) { console.log('Autofixing missing .gitignore file'); const gitignore = - fs.readFileSync('src/bin/plugins/lib/gitignore', {encoding: 'utf8', flag: 'r'}); - fs.writeFileSync(`${pluginPath}/.gitignore`, gitignore); + await fsp.readFile('src/bin/plugins/lib/gitignore', {encoding: 'utf8', flag: 'r'}); + await fsp.writeFile(`${pluginPath}/.gitignore`, gitignore); } } else { let gitignore = - fs.readFileSync(`${pluginPath}/.gitignore`, {encoding: 'utf8', flag: 'r'}); + await fsp.readFile(`${pluginPath}/.gitignore`, {encoding: 'utf8', flag: 'r'}); if (!gitignore.includes('node_modules/')) { console.warn('node_modules/ missing from .gitignore'); if (autoFix) { gitignore += 'node_modules/'; - fs.writeFileSync(`${pluginPath}/.gitignore`, gitignore); + await fsp.writeFile(`${pluginPath}/.gitignore`, gitignore); } } } @@ -321,7 +321,7 @@ fs.readdir(pluginPath, (err, rootFiles) => { 'and should only exist once the plugin has been executed one time.'); if (autoFix) { console.log('Autofixing incorrectly existing .ep_initialized file'); - fs.unlinkSync(`${pluginPath}/.ep_initialized`); + await fsp.unlink(`${pluginPath}/.ep_initialized`); } } @@ -330,16 +330,15 @@ fs.readdir(pluginPath, (err, rootFiles) => { 'your repository.'); if (autoFix) { console.log('Autofixing incorrectly existing npm-debug.log file'); - fs.unlinkSync(`${pluginPath}/npm-debug.log`); + await fsp.unlink(`${pluginPath}/npm-debug.log`); } } if (files.includes('static')) { - fs.readdir(`${pluginPath}/static`, (errRead, staticFiles) => { - if (!staticFiles.includes('tests')) { - console.warn('Test files not found, please create tests. https://github.com/ether/etherpad-lite/wiki/Creating-a-plugin#writing-and-running-front-end-tests-for-your-plugin'); - } - }); + const staticFiles = await fsp.readdir(`${pluginPath}/static`); + if (!staticFiles.includes('tests')) { + console.warn('Test files not found, please create tests. https://github.com/ether/etherpad-lite/wiki/Creating-a-plugin#writing-and-running-front-end-tests-for-your-plugin'); + } } else { console.warn('Test files not found, please create tests. https://github.com/ether/etherpad-lite/wiki/Creating-a-plugin#writing-and-running-front-end-tests-for-your-plugin'); } @@ -374,7 +373,7 @@ fs.readdir(pluginPath, (err, rootFiles) => { env: {...process.env, GIT_INDEX_FILE: '.git/checkPlugin.index'}, stdio: 'inherit', }); - fs.unlinkSync(`${pluginPath}/.git/checkPlugin.index`); + await fsp.unlink(`${pluginPath}/.git/checkPlugin.index`); const commitCmd = [ 'git add -A', @@ -401,4 +400,4 @@ fs.readdir(pluginPath, (err, rootFiles) => { } console.log('Finished'); -}); +})(); From 9a85bce21254413d195e9e924e3bab3430af71c5 Mon Sep 17 00:00:00 2001 From: Richard Hansen Date: Thu, 17 Jun 2021 17:30:38 -0400 Subject: [PATCH 071/446] checkPlugin: Only consider `README{,.md}` (case-insensitive) This avoids false positives such as `README-foo.md`. --- src/bin/plugins/checkPlugin.js | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/bin/plugins/checkPlugin.js b/src/bin/plugins/checkPlugin.js index b7a45f3fb..5488d642b 100755 --- a/src/bin/plugins/checkPlugin.js +++ b/src/bin/plugins/checkPlugin.js @@ -139,7 +139,7 @@ if (autoPush) { let repository; const files = rootFiles.map((f) => { const fl = f.toLowerCase(); - if (fl.includes('readme')) readMeFileName = f; + if (fl === 'readme' || f === 'readme.md') readMeFileName = f; return fl; }); @@ -215,7 +215,7 @@ if (autoPush) { console.warn('Run npm install in the plugin folder and commit the package-lock.json file.'); } } - if (!files.includes('readme') && !files.includes('readme.md')) { + if (!readMeFileName) { console.warn('README.md file not found, please create'); if (autoFix) { console.log('Autofixing missing README.md file'); From 4716975c3740f5712501bc9aa8d440375f468d85 Mon Sep 17 00:00:00 2001 From: Richard Hansen Date: Thu, 17 Jun 2021 17:35:54 -0400 Subject: [PATCH 072/446] checkPlugin: Do case-sensitive filename checks --- src/bin/plugins/checkPlugin.js | 13 ++++--------- 1 file changed, 4 insertions(+), 9 deletions(-) diff --git a/src/bin/plugins/checkPlugin.js b/src/bin/plugins/checkPlugin.js index 5488d642b..b8f4af98b 100755 --- a/src/bin/plugins/checkPlugin.js +++ b/src/bin/plugins/checkPlugin.js @@ -132,16 +132,11 @@ if (autoPush) { } (async () => { - const rootFiles = await fsp.readdir(pluginPath); + const files = await fsp.readdir(pluginPath); // some files we need to know the actual file name. Not compulsory but might help in the future. - let readMeFileName; + const readMeFileName = files.filter((f) => f === 'README' || f === 'README.md')[0]; let repository; - const files = rootFiles.map((f) => { - const fl = f.toLowerCase(); - if (fl === 'readme' || f === 'readme.md') readMeFileName = f; - return fl; - }); if (!files.includes('.git')) throw new Error('No .git folder, aborting'); prepareRepo(); @@ -235,7 +230,7 @@ if (autoPush) { } } - if (!files.includes('contributing') && !files.includes('contributing.md')) { + if (!files.includes('CONTRIBUTING') && !files.includes('CONTRIBUTING.md')) { console.warn('CONTRIBUTING.md file not found, please create'); if (autoFix) { console.log('Autofixing missing CONTRIBUTING.md file, please edit the CONTRIBUTING.md ' + @@ -275,7 +270,7 @@ if (autoPush) { } } - if (!files.includes('license') && !files.includes('license.md')) { + if (!files.includes('LICENSE') && !files.includes('LICENSE.md')) { console.warn('LICENSE.md file not found, please create'); if (autoFix) { console.log('Autofixing missing LICENSE.md file, including Apache 2 license.'); From 3563fc1df9192952a9ea8195edc0e6b248909658 Mon Sep 17 00:00:00 2001 From: Richard Hansen Date: Thu, 9 Dec 2021 20:39:40 -0500 Subject: [PATCH 073/446] checkPlugin: Relax repo checks --- src/bin/plugins/checkPlugin.js | 22 ++++++++++++---------- 1 file changed, 12 insertions(+), 10 deletions(-) diff --git a/src/bin/plugins/checkPlugin.js b/src/bin/plugins/checkPlugin.js index b8f4af98b..05de95ba9 100755 --- a/src/bin/plugins/checkPlugin.js +++ b/src/bin/plugins/checkPlugin.js @@ -86,26 +86,28 @@ const updateDeps = async (parsedPackageJson, key, wantDeps) => { }; const prepareRepo = () => { - let branch = execSync('git symbolic-ref HEAD'); - if (branch !== 'refs/heads/master' && branch !== 'refs/heads/main') { - throw new Error('master/main must be checked out'); - } - branch = branch.replace(/^refs\/heads\//, ''); - execSync('git rev-parse --verify -q HEAD^0 || ' + - `{ echo "Error: no commits on ${branch}" >&2; exit 1; }`); - execSync('git rev-parse --verify @{u}'); // Make sure there's a remote tracking branch. const modified = execSync('git diff-files --name-status'); if (modified !== '') throw new Error(`working directory has modifications:\n${modified}`); const untracked = execSync('git ls-files -o --exclude-standard'); if (untracked !== '') throw new Error(`working directory has untracked files:\n${untracked}`); const indexStatus = execSync('git diff-index --cached --name-status HEAD'); if (indexStatus !== '') throw new Error(`uncommitted staged changes to files:\n${indexStatus}`); - execSync('git pull --ff-only', {stdio: 'inherit'}); - if (execSync('git rev-list @{u}...') !== '') throw new Error('repo contains unpushed commits'); + let br; if (autoCommit) { + br = execSync('git symbolic-ref HEAD'); + if (!br.startsWith('refs/heads/')) throw new Error('detached HEAD'); + br = br.replace(/^refs\/heads\//, ''); + execSync('git rev-parse --verify -q HEAD^0 || ' + + `{ echo "Error: no commits on ${br}" >&2; exit 1; }`); execSync('git config --get user.name'); execSync('git config --get user.email'); } + if (autoPush) { + if (!['master', 'main'].includes(br)) throw new Error('master/main not checked out'); + execSync('git rev-parse --verify @{u}'); + execSync('git pull --ff-only', {stdio: 'inherit'}); + if (execSync('git rev-list @{u}...') !== '') throw new Error('repo contains unpushed commits'); + } }; const checkFile = async (srcFn, dstFn) => { From 2c05de703327484a1659c2f22046ea4bb814cccd Mon Sep 17 00:00:00 2001 From: Richard Hansen Date: Thu, 9 Dec 2021 20:44:58 -0500 Subject: [PATCH 074/446] checkPlugin: Update ESLint dependencies --- src/bin/plugins/checkPlugin.js | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/src/bin/plugins/checkPlugin.js b/src/bin/plugins/checkPlugin.js index 05de95ba9..5340d8157 100755 --- a/src/bin/plugins/checkPlugin.js +++ b/src/bin/plugins/checkPlugin.js @@ -166,14 +166,14 @@ if (autoPush) { } await updateDeps(parsedPackageJSON, 'devDependencies', { - 'eslint': '^7.28.0', - 'eslint-config-etherpad': '^2.0.0', - 'eslint-plugin-cypress': '^2.11.3', + 'eslint': '^7.32.0', + 'eslint-config-etherpad': '^2.0.2', + 'eslint-plugin-cypress': '^2.12.1', 'eslint-plugin-eslint-comments': '^3.2.0', 'eslint-plugin-mocha': '^9.0.0', 'eslint-plugin-node': '^11.1.0', 'eslint-plugin-prefer-arrow': '^1.2.3', - 'eslint-plugin-promise': '^5.1.0', + 'eslint-plugin-promise': '^5.1.1', 'eslint-plugin-you-dont-need-lodash-underscore': '^6.12.0', }); From d81546ad7bbe33f1023ffcb1d87ed02d78b2dbee Mon Sep 17 00:00:00 2001 From: Richard Hansen Date: Thu, 9 Dec 2021 21:32:28 -0500 Subject: [PATCH 075/446] checkPlugin: Delete Travis badge from README.md template --- src/bin/plugins/checkPlugin.js | 21 +-------------------- src/bin/plugins/lib/README.md | 2 -- 2 files changed, 1 insertion(+), 22 deletions(-) diff --git a/src/bin/plugins/checkPlugin.js b/src/bin/plugins/checkPlugin.js index 5340d8157..4aff46c37 100755 --- a/src/bin/plugins/checkPlugin.js +++ b/src/bin/plugins/checkPlugin.js @@ -138,7 +138,6 @@ if (autoPush) { // some files we need to know the actual file name. Not compulsory but might help in the future. const readMeFileName = files.filter((f) => f === 'README' || f === 'README.md')[0]; - let repository; if (!files.includes('.git')) throw new Error('No .git folder, aborting'); prepareRepo(); @@ -155,16 +154,6 @@ if (autoPush) { await fsp.readFile(`${pluginPath}/package.json`, {encoding: 'utf8', flag: 'r'}); const parsedPackageJSON = JSON.parse(packageJSON); - if (!packageJSON.toLowerCase().includes('repository')) { - console.warn('No repository in package.json'); - if (autoFix) { - console.warn('Repository not detected in package.json. Add repository section.'); - } - } else { - // useful for creating README later. - repository = parsedPackageJSON.repository.url; - } - await updateDeps(parsedPackageJSON, 'devDependencies', { 'eslint': '^7.32.0', 'eslint-config-etherpad': '^2.0.2', @@ -220,15 +209,7 @@ if (autoPush) { let readme = await fsp.readFile('src/bin/plugins/lib/README.md', {encoding: 'utf8', flag: 'r'}); readme = readme.replace(/\[plugin_name\]/g, pluginName); - if (repository) { - const org = repository.split('/')[3]; - const name = repository.split('/')[4]; - readme = readme.replace(/\[org_name\]/g, org); - readme = readme.replace(/\[repo_url\]/g, name); - await fsp.writeFile(`${pluginPath}/README.md`, readme); - } else { - console.warn('Unable to find repository in package.json, aborting.'); - } + await fsp.writeFile(`${pluginPath}/README.md`, readme); } } diff --git a/src/bin/plugins/lib/README.md b/src/bin/plugins/lib/README.md index e17a23ed1..e7bdef01d 100755 --- a/src/bin/plugins/lib/README.md +++ b/src/bin/plugins/lib/README.md @@ -1,5 +1,3 @@ -[![Travis (.com)](https://api.travis-ci.com/[org_name]/[repo_url].svg?branch=develop)](https://travis-ci.com/github/[org_name]/[repo_url]) - # My awesome plugin README example Explain what your plugin does and who it's useful for. From 5915c2243da3e16f3636009e4aa3785337376bb2 Mon Sep 17 00:00:00 2001 From: Richard Hansen Date: Thu, 9 Dec 2021 21:48:59 -0500 Subject: [PATCH 076/446] checkPlugin: Redo README.md and LICENSE --- src/bin/plugins/checkPlugin.js | 29 +++-- src/bin/plugins/lib/LICENSE | 201 +++++++++++++++++++++++++++++++++ src/bin/plugins/lib/LICENSE.md | 13 --- src/bin/plugins/lib/README.md | 45 +++++--- 4 files changed, 246 insertions(+), 42 deletions(-) create mode 100644 src/bin/plugins/lib/LICENSE delete mode 100755 src/bin/plugins/lib/LICENSE.md diff --git a/src/bin/plugins/checkPlugin.js b/src/bin/plugins/checkPlugin.js index 4aff46c37..82e29df94 100755 --- a/src/bin/plugins/checkPlugin.js +++ b/src/bin/plugins/checkPlugin.js @@ -201,15 +201,21 @@ if (autoPush) { console.warn('Run npm install in the plugin folder and commit the package-lock.json file.'); } } + + const fillTemplate = async (templateFilename, outputFilename) => { + const contents = (await fsp.readFile(templateFilename, 'utf8')) + .replace(/\[name of copyright owner\]/g, execSync('git config user.name')) + .replace(/\[plugin_name\]/g, pluginName) + .replace(/\[yyyy\]/g, new Date().getFullYear()); + await fsp.writeFile(outputFilename, contents); + }; + if (!readMeFileName) { console.warn('README.md file not found, please create'); if (autoFix) { console.log('Autofixing missing README.md file'); console.log('please edit the README.md file further to include plugin specific details.'); - let readme = - await fsp.readFile('src/bin/plugins/lib/README.md', {encoding: 'utf8', flag: 'r'}); - readme = readme.replace(/\[plugin_name\]/g, pluginName); - await fsp.writeFile(`${pluginPath}/README.md`, readme); + await fillTemplate('src/bin/plugins/lib/README.md', `${pluginPath}/README.md`); } } @@ -218,10 +224,7 @@ if (autoPush) { if (autoFix) { console.log('Autofixing missing CONTRIBUTING.md file, please edit the CONTRIBUTING.md ' + 'file further to include plugin specific details.'); - let contributing = - await fsp.readFile('src/bin/plugins/lib/CONTRIBUTING.md', {encoding: 'utf8', flag: 'r'}); - contributing = contributing.replace(/\[plugin_name\]/g, pluginName); - await fsp.writeFile(`${pluginPath}/CONTRIBUTING.md`, contributing); + await fillTemplate('src/bin/plugins/lib/CONTRIBUTING.md', `${pluginPath}/CONTRIBUTING.md`); } } @@ -254,14 +257,10 @@ if (autoPush) { } if (!files.includes('LICENSE') && !files.includes('LICENSE.md')) { - console.warn('LICENSE.md file not found, please create'); + console.warn('LICENSE file not found, please create'); if (autoFix) { - console.log('Autofixing missing LICENSE.md file, including Apache 2 license.'); - let license = - await fsp.readFile('src/bin/plugins/lib/LICENSE.md', {encoding: 'utf8', flag: 'r'}); - license = license.replace('[yyyy]', new Date().getFullYear()); - license = license.replace('[name of copyright owner]', execSync('git config user.name')); - await fsp.writeFile(`${pluginPath}/LICENSE.md`, license); + console.log('Autofixing missing LICENSE file (Apache 2.0).'); + await fsp.copyFile('src/bin/plugins/lib/LICENSE', `${pluginPath}/LICENSE`); } } diff --git a/src/bin/plugins/lib/LICENSE b/src/bin/plugins/lib/LICENSE new file mode 100644 index 000000000..261eeb9e9 --- /dev/null +++ b/src/bin/plugins/lib/LICENSE @@ -0,0 +1,201 @@ + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. diff --git a/src/bin/plugins/lib/LICENSE.md b/src/bin/plugins/lib/LICENSE.md deleted file mode 100755 index 004c62e1b..000000000 --- a/src/bin/plugins/lib/LICENSE.md +++ /dev/null @@ -1,13 +0,0 @@ -Copyright [yyyy] [name of copyright owner] - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. diff --git a/src/bin/plugins/lib/README.md b/src/bin/plugins/lib/README.md index e7bdef01d..2c50b538e 100755 --- a/src/bin/plugins/lib/README.md +++ b/src/bin/plugins/lib/README.md @@ -1,30 +1,47 @@ -# My awesome plugin README example -Explain what your plugin does and who it's useful for. +# [plugin_name] + +TODO: Describe the plugin. ## Example animated gif of usage if appropriate + ![screenshot](https://user-images.githubusercontent.com/220864/99979953-97841d80-2d9f-11eb-9782-5f65817c58f4.PNG) -## Installing +## Installation -``` +From the Etherpad working directory, run: + +```shell npm install --no-save --legacy-peer-deps [plugin_name] ``` -or Use the Etherpad ``/admin`` interface. +Or, install from Etherpad's `/admin/plugins` page. -## Settings -Document settings if any +## Configuration + +TODO ## Testing -Document how to run backend / frontend tests. -### Frontend +To run the backend tests, run the following from the Etherpad working directory: -Visit http://whatever/tests/frontend/ to run the frontend tests. +```shell +(cd src && npm test) +``` -### backend +To run the frontend tests, visit: http://localhost:9001/tests/frontend/ -Type ``cd src && npm run test`` to run the backend tests. +## Copyright and License -## LICENSE -Apache 2.0 +Copyright © [yyyy] [name of copyright owner] +and the [plugin_name] authors and contributors + +Licensed under the [Apache License, Version 2.0](LICENSE) (the "License"); you +may not use this file except in compliance with the License. You may obtain a +copy of the License at + +http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software distributed +under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR +CONDITIONS OF ANY KIND, either express or implied. See the License for the +specific language governing permissions and limitations under the License. From 1fe01c66fd848cd97d2600a7f0ca1865bf920a9c Mon Sep 17 00:00:00 2001 From: Richard Hansen Date: Fri, 10 Dec 2021 15:03:50 -0500 Subject: [PATCH 077/446] getCorePlugins.sh: Various improvements * Factor out plugin query. * Make idempotent. * Improve logging. * Install by symlinking to a parallel directory rather than cloning into `etherpad-lite/node_modules`. --- src/bin/plugins/getCorePlugins.sh | 43 ++++++++++++++++++++++++++--- src/bin/plugins/listOfficialPlugins | 14 ++++++++++ 2 files changed, 53 insertions(+), 4 deletions(-) create mode 100755 src/bin/plugins/listOfficialPlugins diff --git a/src/bin/plugins/getCorePlugins.sh b/src/bin/plugins/getCorePlugins.sh index e8ce68b21..85552ab14 100755 --- a/src/bin/plugins/getCorePlugins.sh +++ b/src/bin/plugins/getCorePlugins.sh @@ -1,4 +1,39 @@ -cd node_modules/ -GHUSER=ether; curl "https://api.github.com/users/$GHUSER/repos?per_page=100" | grep -o 'git@[^"]*' | grep /ep_ | xargs -L1 git clone -GHUSER=ether; curl "https://api.github.com/users/$GHUSER/repos?per_page=100&page=2&" | grep -o 'git@[^"]*' | grep /ep_ | xargs -L1 git clone -GHUSER=ether; curl "https://api.github.com/users/$GHUSER/repos?per_page=100&page=3&" | grep -o 'git@[^"]*' | grep /ep_ | xargs -L1 git clone +#!/bin/sh + +set -e + +newline=' +' + +pecho () { printf %s\\n "$*"; } +log () { pecho "$@"; } +error () { log "ERROR: $@" >&2; } +fatal () { error "$@"; exit 1; } + +mydir=$(cd "${0%/*}" && pwd -P) || exit 1 +cd "${mydir}/../../.." +pdir=$(cd .. && pwd -P) || exit 1 + +plugins=$("${mydir}/listOfficialPlugins") || exit 1 +for d in ${plugins}; do + log "============================================================" + log "${d}" + log "============================================================" + fd=${pdir}/${d} + repo=git@github.com:ether/${plugin}.git + [ -d "${fd}" ] || { + log "Cloning ${repo} to ${fd}..." + (cd "${pdir}" && git clone "${repo}" "${d}") || exit 1 + } || exit 1 + log "Fetching latest commits..." + (cd "${fd}" && git pull --ff-only) || exit 1 + log "Getting plugin name..." + pn=$(cd "${fd}" && npx -c 'printf %s\\n "${npm_package_name}"') || exit 1 + [ -n "${pn}" ] || fatal "Unable to determine plugin name for ${d}" + md=node_modules/${pn} + [ -d "${md}" ] || { + log "Installing plugin to ${md}..." + ln -s ../../"${d}" "${md}" + } || exit 1 + [ "${md}" -ef "${fd}" ] || fatal "${md} is not a symlink to ${fd}" +done diff --git a/src/bin/plugins/listOfficialPlugins b/src/bin/plugins/listOfficialPlugins new file mode 100755 index 000000000..322ad5d3b --- /dev/null +++ b/src/bin/plugins/listOfficialPlugins @@ -0,0 +1,14 @@ +#!/bin/sh +set -e +newline=' +' +mydir=$(cd "${0%/*}" && pwd -P) || exit 1 +cd "${mydir}/../../.." +pdir=$(cd .. && pwd -P) || exit 1 +plugins= +for p in "" "&page=2" "&page=3"; do + curlOut=$(curl "https://api.github.com/users/ether/repos?per_page=100${p}") || exit 1 + plugins=${plugins}${newline}$(printf %s\\n "${curlOut}" \ + | sed -n -e 's;.*git@github.com:ether/\(ep_[^"]*\)\.git.*;\1;p'); +done +printf %s\\n "${plugins}" | sort -u | grep -v '^[[:space:]]*$' From 2cae414473373d882b43943c697d804a88396478 Mon Sep 17 00:00:00 2001 From: "translatewiki.net" Date: Mon, 13 Dec 2021 13:03:50 +0100 Subject: [PATCH 078/446] Localisation updates from https://translatewiki.net. --- src/locales/vec.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/locales/vec.json b/src/locales/vec.json index b925baf73..97892523e 100644 --- a/src/locales/vec.json +++ b/src/locales/vec.json @@ -27,7 +27,7 @@ "pad.colorpicker.cancel": "Descançełare", "pad.loading": "Drio cargar...", "pad.noCookie": "El cookie no el xé sta catà. Cosenti i cookie n'tel to navegadore web.", - "timeslider.month.january": "Xenaro", + "timeslider.month.january": "Zenaro", "timeslider.month.march": "Marso", "timeslider.month.april": "Apriłe", "timeslider.month.may": "Majo", From dbacc73c369a9583a4ffeccc83d7aed3aeec22b2 Mon Sep 17 00:00:00 2001 From: Richard Hansen Date: Sun, 12 Dec 2021 03:46:58 -0500 Subject: [PATCH 079/446] tests: Basic USER_CHANGES backend tests --- src/tests/backend/common.js | 15 ++++ src/tests/backend/specs/messages.js | 103 ++++++++++++++++++++++++++++ 2 files changed, 118 insertions(+) create mode 100644 src/tests/backend/specs/messages.js diff --git a/src/tests/backend/common.js b/src/tests/backend/common.js index 793828ac7..89f635012 100644 --- a/src/tests/backend/common.js +++ b/src/tests/backend/common.js @@ -184,3 +184,18 @@ exports.handshake = async (socket, padId) => { logger.debug('received CLIENT_VARS message'); return msg; }; + +const alphabet = 'abcdefghijklmnopqrstuvwxyz'; + +/** + * Generates a random string. + * + * @param {number} [len] - The desired length of the generated string. + * @param {string} [charset] - Characters to pick from. + * @returns {string} + */ +exports.randomString = (len = 10, charset = `${alphabet}${alphabet.toUpperCase()}0123456789`) => { + let ret = ''; + while (ret.length < len) ret += charset[Math.floor(Math.random() * charset.length)]; + return ret; +}; diff --git a/src/tests/backend/specs/messages.js b/src/tests/backend/specs/messages.js new file mode 100644 index 000000000..7e58ca5e7 --- /dev/null +++ b/src/tests/backend/specs/messages.js @@ -0,0 +1,103 @@ +'use strict'; + +const AttributePool = require('../../../static/js/AttributePool'); +const assert = require('assert').strict; +const common = require('../common'); +const padManager = require('../../../node/db/PadManager'); + +describe(__filename, function () { + let agent; + let pad; + let padId; + let rev; + let socket; + + before(async function () { + agent = await common.init(); + }); + + beforeEach(async function () { + padId = common.randomString(); + assert(!await padManager.doesPadExist(padId)); + pad = await padManager.getPad(padId, ''); + assert.equal(pad.text(), '\n'); + const res = await agent.get(`/p/${padId}`).expect(200); + socket = await common.connect(res); + const {type, data: clientVars} = await common.handshake(socket, padId); + assert.equal(type, 'CLIENT_VARS'); + rev = clientVars.collab_client_vars.rev; + }); + + afterEach(async function () { + if (socket != null) socket.close(); + socket = null; + if (pad != null) await pad.remove(); + pad = null; + }); + + describe('USER_CHANGES', function () { + const sendUserChanges = (changeset, apool = new AttributePool()) => { + socket.json.send({ + type: 'COLLABROOM', + component: 'pad', + data: { + type: 'USER_CHANGES', + baseRev: rev, + changeset, + apool: new AttributePool(), + }, + }); + }; + const assertAccepted = async (wantRev) => { + const msg = await common.waitForSocketEvent(socket, 'message'); + assert.deepEqual(msg, { + type: 'COLLABROOM', + data: { + type: 'ACCEPT_COMMIT', + newRev: wantRev, + }, + }); + rev = wantRev; + }; + const assertRejected = async () => { + const msg = await common.waitForSocketEvent(socket, 'message'); + assert.deepEqual(msg, {disconnect: 'badChangeset'}); + }; + + it('changes are applied', async function () { + sendUserChanges('Z:1>5+5$hello'); + await assertAccepted(rev + 1); + assert.equal(pad.text(), 'hello\n'); + }); + + it('bad changeset is rejected', async function () { + sendUserChanges('this is not a valid changeset'); + await assertRejected(); + }); + + it('retransmission is rejected', async function () { + sendUserChanges('Z:1>5+5$hello'); + await assertAccepted(rev + 1); + --rev; + sendUserChanges('Z:1>5+5$hello'); + await assertRejected(); + assert.equal(pad.text(), 'hello\n'); + }); + + it('identity changeset is accepted', async function () { + sendUserChanges('Z:1>5+5$hello'); + await assertAccepted(rev + 1); + sendUserChanges('Z:6>0$'); + await assertAccepted(rev + 1); + assert.equal(pad.text(), 'hello\n'); + }); + + it('non-identity changeset with no net change is accepted', async function () { + sendUserChanges('Z:1>5+5$hello'); + await assertAccepted(rev + 1); + sendUserChanges('Z:6>0-5+5$hello'); + await assertAccepted(rev + 1); + assert.equal(pad.text(), 'hello\n'); + }); + }); +}); From c05ee7ce72a18459992dc689375fdff544e882da Mon Sep 17 00:00:00 2001 From: Richard Hansen Date: Sun, 12 Dec 2021 18:03:16 -0500 Subject: [PATCH 080/446] PadMessageHandler: Move `ACCEPT_COMMIT` after changeset save --- src/node/handler/PadMessageHandler.js | 43 ++++++++++++++++----------- 1 file changed, 25 insertions(+), 18 deletions(-) diff --git a/src/node/handler/PadMessageHandler.js b/src/node/handler/PadMessageHandler.js index 0225b00d1..2c8c7ad77 100644 --- a/src/node/handler/PadMessageHandler.js +++ b/src/node/handler/PadMessageHandler.js @@ -640,6 +640,7 @@ const handleUserChanges = async (socket, message) => { } await pad.appendRevision(rebasedChangeset, thisSession.author); + assert.equal(pad.getHeadRevisionNumber(), r + 1); const correctionChangeset = _correctMarkersInPad(pad.atext, pad.pool); if (correctionChangeset) { @@ -652,6 +653,17 @@ const handleUserChanges = async (socket, message) => { await pad.appendRevision(nlChangeset); } + // The client assumes that ACCEPT_COMMIT and NEW_CHANGES messages arrive in order. Make sure we + // have already sent any previous ACCEPT_COMMIT and NEW_CHANGES messages. + assert.equal(thisSession.rev, r); + socket.json.send({ + type: 'COLLABROOM', + data: { + type: 'ACCEPT_COMMIT', + newRev: ++thisSession.rev, + }, + }); + thisSession.time = await pad.getRevisionDate(thisSession.rev); await exports.updatePadClients(pad); } catch (err) { socket.json.send({disconnect: 'badChangeset'}); @@ -697,24 +709,19 @@ exports.updatePadClients = async (pad) => { const revChangeset = revision.changeset; const currentTime = revision.meta.timestamp; - let msg; - if (author === sessioninfo.author) { - msg = {type: 'COLLABROOM', data: {type: 'ACCEPT_COMMIT', newRev: r}}; - } else { - const forWire = Changeset.prepareForWire(revChangeset, pad.pool); - msg = { - type: 'COLLABROOM', - data: { - type: 'NEW_CHANGES', - newRev: r, - changeset: forWire.translated, - apool: forWire.pool, - author, - currentTime, - timeDelta: currentTime - sessioninfo.time, - }, - }; - } + const forWire = Changeset.prepareForWire(revChangeset, pad.pool); + const msg = { + type: 'COLLABROOM', + data: { + type: 'NEW_CHANGES', + newRev: r, + changeset: forWire.translated, + apool: forWire.pool, + author, + currentTime, + timeDelta: currentTime - sessioninfo.time, + }, + }; try { socket.json.send(msg); } catch (err) { From 56b767142264bc692defe5b0e0bcf3bac3bc083e Mon Sep 17 00:00:00 2001 From: Richard Hansen Date: Sun, 12 Dec 2021 18:10:33 -0500 Subject: [PATCH 081/446] Pad: Return new rev number from `appendRevision()` --- src/node/db/Pad.js | 1 + src/node/handler/PadMessageHandler.js | 15 +++++---------- 2 files changed, 6 insertions(+), 10 deletions(-) diff --git a/src/node/db/Pad.js b/src/node/db/Pad.js index 0379f512a..664ad55ff 100644 --- a/src/node/db/Pad.js +++ b/src/node/db/Pad.js @@ -119,6 +119,7 @@ Pad.prototype.appendRevision = async function (aChangeset, author) { } await Promise.all(p); + return newRev; }; // save all attributes to the database diff --git a/src/node/handler/PadMessageHandler.js b/src/node/handler/PadMessageHandler.js index 2c8c7ad77..cdc795a3d 100644 --- a/src/node/handler/PadMessageHandler.js +++ b/src/node/handler/PadMessageHandler.js @@ -639,8 +639,8 @@ const handleUserChanges = async (socket, message) => { `${Changeset.oldLen(rebasedChangeset)} to document of length ${prevText.length}`); } - await pad.appendRevision(rebasedChangeset, thisSession.author); - assert.equal(pad.getHeadRevisionNumber(), r + 1); + const newRev = await pad.appendRevision(rebasedChangeset, thisSession.author); + assert.equal(newRev, r + 1); const correctionChangeset = _correctMarkersInPad(pad.atext, pad.pool); if (correctionChangeset) { @@ -656,14 +656,9 @@ const handleUserChanges = async (socket, message) => { // The client assumes that ACCEPT_COMMIT and NEW_CHANGES messages arrive in order. Make sure we // have already sent any previous ACCEPT_COMMIT and NEW_CHANGES messages. assert.equal(thisSession.rev, r); - socket.json.send({ - type: 'COLLABROOM', - data: { - type: 'ACCEPT_COMMIT', - newRev: ++thisSession.rev, - }, - }); - thisSession.time = await pad.getRevisionDate(thisSession.rev); + socket.json.send({type: 'COLLABROOM', data: {type: 'ACCEPT_COMMIT', newRev}}); + thisSession.rev = newRev; + thisSession.time = await pad.getRevisionDate(newRev); await exports.updatePadClients(pad); } catch (err) { socket.json.send({disconnect: 'badChangeset'}); From a370cfa5c69eea0c70090398b1a2f101d9bc2afa Mon Sep 17 00:00:00 2001 From: Richard Hansen Date: Sat, 11 Dec 2021 20:03:35 -0500 Subject: [PATCH 082/446] Pad: Don't create no-op revisions --- src/node/db/Pad.js | 6 ++++-- src/node/handler/PadMessageHandler.js | 6 ++++-- src/node/utils/ImportHtml.js | 2 +- src/static/js/collab_client.js | 6 ++++-- src/tests/backend/specs/api/pad.js | 2 +- src/tests/backend/specs/messages.js | 8 ++++---- 6 files changed, 18 insertions(+), 12 deletions(-) diff --git a/src/node/db/Pad.js b/src/node/db/Pad.js index 664ad55ff..abca75102 100644 --- a/src/node/db/Pad.js +++ b/src/node/db/Pad.js @@ -82,6 +82,9 @@ Pad.prototype.appendRevision = async function (aChangeset, author) { } const newAText = Changeset.applyToAText(aChangeset, this.atext, this.pool); + if (newAText.text === this.atext.text && newAText.attribs === this.atext.attribs) { + return this.head; + } Changeset.copyAText(newAText, this.atext); const newRev = ++this.head; @@ -268,8 +271,7 @@ Pad.prototype.setText = async function (newText) { changeset = Changeset.makeSplice(oldText, 0, oldText.length - 1, newText); } - // append the changeset - if (newText !== oldText) await this.appendRevision(changeset); + await this.appendRevision(changeset); }; Pad.prototype.appendText = async function (newText) { diff --git a/src/node/handler/PadMessageHandler.js b/src/node/handler/PadMessageHandler.js index cdc795a3d..f50f7c331 100644 --- a/src/node/handler/PadMessageHandler.js +++ b/src/node/handler/PadMessageHandler.js @@ -640,7 +640,9 @@ const handleUserChanges = async (socket, message) => { } const newRev = await pad.appendRevision(rebasedChangeset, thisSession.author); - assert.equal(newRev, r + 1); + // The head revision will either stay the same or increase by 1 depending on whether the + // changeset has a net effect. + assert([r, r + 1].includes(newRev)); const correctionChangeset = _correctMarkersInPad(pad.atext, pad.pool); if (correctionChangeset) { @@ -658,7 +660,7 @@ const handleUserChanges = async (socket, message) => { assert.equal(thisSession.rev, r); socket.json.send({type: 'COLLABROOM', data: {type: 'ACCEPT_COMMIT', newRev}}); thisSession.rev = newRev; - thisSession.time = await pad.getRevisionDate(newRev); + if (newRev !== r) thisSession.time = await pad.getRevisionDate(newRev); await exports.updatePadClients(pad); } catch (err) { socket.json.send({disconnect: 'badChangeset'}); diff --git a/src/node/utils/ImportHtml.js b/src/node/utils/ImportHtml.js index 059d57af6..12a99ef79 100644 --- a/src/node/utils/ImportHtml.js +++ b/src/node/utils/ImportHtml.js @@ -85,5 +85,5 @@ exports.setPadHTML = async (pad, html) => { apiLogger.debug(`The changeset: ${theChangeset}`); await pad.setText('\n'); - if (!Changeset.isIdentity(theChangeset)) await pad.appendRevision(theChangeset); + await pad.appendRevision(theChangeset); }; diff --git a/src/static/js/collab_client.js b/src/static/js/collab_client.js index 849fff5fc..38dc22af4 100644 --- a/src/static/js/collab_client.js +++ b/src/static/js/collab_client.js @@ -207,8 +207,10 @@ const getCollabClient = (ace2editor, serverVars, initialUserInfo, options, _pad) }); } else if (msg.type === 'ACCEPT_COMMIT') { serverMessageTaskQueue.enqueue(() => { - const newRev = msg.newRev; - if (newRev !== (rev + 1)) { + const {newRev} = msg; + // newRev will equal rev if the changeset has no net effect (identity changeset, or removing + // and re-adding the same characters with the same attributes). + if (![rev, rev + 1].includes(newRev)) { window.console.warn(`bad message revision on ACCEPT_COMMIT: ${newRev} not ${rev + 1}`); // setChannelState("DISCONNECTED", "badmessage_acceptcommit"); return; diff --git a/src/tests/backend/specs/api/pad.js b/src/tests/backend/specs/api/pad.js index 41c30b8a0..64f10f012 100644 --- a/src/tests/backend/specs/api/pad.js +++ b/src/tests/backend/specs/api/pad.js @@ -278,7 +278,7 @@ describe(__filename, function () { const res = await agent.post(endPoint('setText')) .send({ padID: testPadId, - text: 'testTextTwo', + text: 'testTextThree', }) .expect(200) .expect('Content-Type', /json/); diff --git a/src/tests/backend/specs/messages.js b/src/tests/backend/specs/messages.js index 7e58ca5e7..55d3c7a19 100644 --- a/src/tests/backend/specs/messages.js +++ b/src/tests/backend/specs/messages.js @@ -84,19 +84,19 @@ describe(__filename, function () { assert.equal(pad.text(), 'hello\n'); }); - it('identity changeset is accepted', async function () { + it('identity changeset is accepted, has no effect', async function () { sendUserChanges('Z:1>5+5$hello'); await assertAccepted(rev + 1); sendUserChanges('Z:6>0$'); - await assertAccepted(rev + 1); + await assertAccepted(rev); assert.equal(pad.text(), 'hello\n'); }); - it('non-identity changeset with no net change is accepted', async function () { + it('non-identity changeset with no net change is accepted, has no effect', async function () { sendUserChanges('Z:1>5+5$hello'); await assertAccepted(rev + 1); sendUserChanges('Z:6>0-5+5$hello'); - await assertAccepted(rev + 1); + await assertAccepted(rev); assert.equal(pad.text(), 'hello\n'); }); }); From cff089e54e18f39b0bddd127e0c425849800d813 Mon Sep 17 00:00:00 2001 From: Richard Hansen Date: Sun, 12 Dec 2021 18:19:36 -0500 Subject: [PATCH 083/446] PadMessageHandler: Accept retransmissions of USER_CHANGES --- src/node/handler/PadMessageHandler.js | 14 +++++--------- src/static/js/collab_client.js | 5 +++-- src/tests/backend/specs/messages.js | 4 ++-- 3 files changed, 10 insertions(+), 13 deletions(-) diff --git a/src/node/handler/PadMessageHandler.js b/src/node/handler/PadMessageHandler.js index f50f7c331..3460983bd 100644 --- a/src/node/handler/PadMessageHandler.js +++ b/src/node/handler/PadMessageHandler.js @@ -615,19 +615,15 @@ const handleUserChanges = async (socket, message) => { // Update the changeset so that it can be applied to the latest revision. while (r < pad.getHeadRevisionNumber()) { r++; - - const c = await pad.getRevisionChangeset(r); - + const {changeset: c, meta: {author: authorId}} = await pad.getRevision(r); + if (changeset === c && thisSession.author === authorId) { + // Assume this is a retransmission of an already applied changeset. + rebasedChangeset = Changeset.identity(Changeset.unpack(changeset).oldLen); + } // At this point, both "c" (from the pad) and "changeset" (from the // client) are relative to revision r - 1. The follow function // rebases "changeset" so that it is relative to revision r // and can be applied after "c". - - // a changeset can be based on an old revision with the same changes in it - // prevent eplite from accepting it TODO: better send the client a NEW_CHANGES - // of that revision - if (baseRev + 1 === r && c === changeset) throw new Error('Changeset already accepted'); - rebasedChangeset = Changeset.follow(c, rebasedChangeset, false, pad.pool); } diff --git a/src/static/js/collab_client.js b/src/static/js/collab_client.js index 38dc22af4..74bc66f9f 100644 --- a/src/static/js/collab_client.js +++ b/src/static/js/collab_client.js @@ -208,8 +208,9 @@ const getCollabClient = (ace2editor, serverVars, initialUserInfo, options, _pad) } else if (msg.type === 'ACCEPT_COMMIT') { serverMessageTaskQueue.enqueue(() => { const {newRev} = msg; - // newRev will equal rev if the changeset has no net effect (identity changeset, or removing - // and re-adding the same characters with the same attributes). + // newRev will equal rev if the changeset has no net effect (identity changeset, removing + // and re-adding the same characters with the same attributes, or retransmission of an + // already applied changeset). if (![rev, rev + 1].includes(newRev)) { window.console.warn(`bad message revision on ACCEPT_COMMIT: ${newRev} not ${rev + 1}`); // setChannelState("DISCONNECTED", "badmessage_acceptcommit"); diff --git a/src/tests/backend/specs/messages.js b/src/tests/backend/specs/messages.js index 55d3c7a19..4c9f7e66c 100644 --- a/src/tests/backend/specs/messages.js +++ b/src/tests/backend/specs/messages.js @@ -75,12 +75,12 @@ describe(__filename, function () { await assertRejected(); }); - it('retransmission is rejected', async function () { + it('retransmission is accepted, has no effect', async function () { sendUserChanges('Z:1>5+5$hello'); await assertAccepted(rev + 1); --rev; sendUserChanges('Z:1>5+5$hello'); - await assertRejected(); + await assertAccepted(rev + 1); assert.equal(pad.text(), 'hello\n'); }); From d94f3801410cd47bbaa93a630c8f4e11ee08e1dc Mon Sep 17 00:00:00 2001 From: Richard Hansen Date: Sun, 12 Dec 2021 18:56:32 -0500 Subject: [PATCH 084/446] API: Fix race conditions in `setText`, `appendText`, `restoreRevision` --- CHANGELOG.md | 2 ++ src/node/db/API.js | 18 ++++++------------ 2 files changed, 8 insertions(+), 12 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 8d044260d..14653a33c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -13,6 +13,8 @@ * Fixed a potential attribute pool corruption bug with `copyPadWithoutHistory`. * Mappings created by the `createGroupIfNotExistsFor` HTTP API are now removed from the database when the group is deleted. +* Fixed race conditions in the `setText`, `appendText`, and `restoreRevision` + functions (HTTP API). #### For plugin authors diff --git a/src/node/db/API.js b/src/node/db/API.js index 49d6b9cef..040abf5a6 100644 --- a/src/node/db/API.js +++ b/src/node/db/API.js @@ -201,10 +201,8 @@ exports.setText = async (padID, text) => { // get the pad const pad = await getPadSafe(padID, true); - await Promise.all([ - pad.setText(text), - padMessageHandler.updatePadClients(pad), - ]); + await pad.setText(text); + await padMessageHandler.updatePadClients(pad); }; /** @@ -223,10 +221,8 @@ exports.appendText = async (padID, text) => { } const pad = await getPadSafe(padID, true); - await Promise.all([ - pad.appendText(text), - padMessageHandler.updatePadClients(pad), - ]); + await pad.appendText(text); + await padMessageHandler.updatePadClients(pad); }; /** @@ -559,10 +555,8 @@ exports.restoreRevision = async (padID, rev) => { const changeset = builder.toString(); - await Promise.all([ - pad.appendRevision(changeset), - padMessageHandler.updatePadClients(pad), - ]); + await pad.appendRevision(changeset); + await padMessageHandler.updatePadClients(pad); }; /** From 3693a0574f5b4cd77eb6bdd3f3e6f029b0af5a45 Mon Sep 17 00:00:00 2001 From: snyk-bot Date: Mon, 13 Dec 2021 10:40:31 +0000 Subject: [PATCH 085/446] fix: upgrade jsdom from 18.1.0 to 18.1.1 Snyk has created this PR to upgrade jsdom from 18.1.0 to 18.1.1. See this package in npm: https://www.npmjs.com/package/jsdom See this project in Snyk: https://app.snyk.io/org/johnmclear/project/d9a12bfb-7ccd-443f-9e22-f30d339cc8c5?utm_source=github&utm_medium=referral&page=upgrade-pr --- src/package-lock.json | 18 +++++++++--------- src/package.json | 2 +- 2 files changed, 10 insertions(+), 10 deletions(-) diff --git a/src/package-lock.json b/src/package-lock.json index 8c983062f..baefcb422 100644 --- a/src/package-lock.json +++ b/src/package-lock.json @@ -3226,9 +3226,9 @@ "integrity": "sha1-peZUwuWi3rXyAdls77yoDA7y9RM=" }, "jsdom": { - "version": "18.1.0", - "resolved": "https://registry.npmjs.org/jsdom/-/jsdom-18.1.0.tgz", - "integrity": "sha512-q6QFAfSGLEUqRJ+GCV6vn6ItZCMARWh1d33wiJZPxc+wMNw7HK71JPmQ4C2lIZAsBH8TiJu4uplach/UcrC6bQ==", + "version": "18.1.1", + "resolved": "https://registry.npmjs.org/jsdom/-/jsdom-18.1.1.tgz", + "integrity": "sha512-NmJQbjQ/gpS/1at/ce3nCx89HbXL/f5OcenBe8wU1Eik0ROhyUc3LtmG3567dEHAGXkN8rmILW/qtCOPxPHQJw==", "requires": { "abab": "^2.0.5", "acorn": "^8.5.0", @@ -3265,9 +3265,9 @@ "integrity": "sha512-XCuKFP5PS55gnMVu3dty8KPatLqUoy/ZYzDzAGCQ8JNFCkLXzmI7vNHCR+XpbZaMWQK/vQubr7PkYq8g470J/A==" }, "debug": { - "version": "4.3.2", - "resolved": "https://registry.npmjs.org/debug/-/debug-4.3.2.tgz", - "integrity": "sha512-mOp8wKcvj7XxC78zLgw/ZA+6TSgkoE2C/ienthhRD298T7UNwAg9diBpLRxC0mOezLl4B0xV7M0cCO6P/O0Xhw==", + "version": "4.3.3", + "resolved": "https://registry.npmjs.org/debug/-/debug-4.3.3.tgz", + "integrity": "sha512-/zxw5+vh1Tfv+4Qn7a5nsbcJKPaSvCDhojn6FEl9vupwK2VCSDtEiEtqr8DFtzYFOdz63LBkxec7DYuc2jon6Q==", "requires": { "ms": "2.1.2" } @@ -3288,9 +3288,9 @@ "integrity": "sha512-sGkPx+VjMtmA6MX27oA4FBFELFCZZ4S4XqeGOXCv68tT+jb3vk/RyaKWP0PTKyWtmLSM0b+adUTEvbs1PEaH2w==" }, "ws": { - "version": "8.2.3", - "resolved": "https://registry.npmjs.org/ws/-/ws-8.2.3.tgz", - "integrity": "sha512-wBuoj1BDpC6ZQ1B7DWQBYVLphPWkm8i9Y0/3YdHjHKHiohOJ1ws+3OccDWtH+PoC9DZD5WOTrJvNbWvjS6JWaA==" + "version": "8.3.0", + "resolved": "https://registry.npmjs.org/ws/-/ws-8.3.0.tgz", + "integrity": "sha512-Gs5EZtpqZzLvmIM59w4igITU57lrtYVFneaa434VROv4thzJyV6UjIL3D42lslWlI+D4KzLYnxSwtfuiO79sNw==" } } }, diff --git a/src/package.json b/src/package.json index 4a70eefce..4daac1c46 100644 --- a/src/package.json +++ b/src/package.json @@ -45,7 +45,7 @@ "formidable": "^1.2.6", "http-errors": "^1.8.1", "js-cookie": "^3.0.1", - "jsdom": "^18.1.0", + "jsdom": "^18.1.1", "jsonminify": "0.4.1", "languages4translatewiki": "0.1.3", "lodash.clonedeep": "4.5.0", From 10e2b09b96ee92b365d1bb8be6112e13b16ffa79 Mon Sep 17 00:00:00 2001 From: Robert Geislinger Date: Sat, 11 Dec 2021 21:27:17 +0100 Subject: [PATCH 086/446] Update http_api.md The current version is 1.2.15 or bigger if you look at e.g. copyPadWithoutHistory --- doc/api/http_api.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/api/http_api.md b/doc/api/http_api.md index 9cab7f56b..bf3be2755 100644 --- a/doc/api/http_api.md +++ b/doc/api/http_api.md @@ -65,7 +65,7 @@ Portal submits content into new blog post ## Usage ### API version -The latest version is `1.2.14` +The latest version is `1.2.15` The current version can be queried via /api. From a6bf7816ced632b51461e9e75dd915e0d06d42f2 Mon Sep 17 00:00:00 2001 From: Richard Hansen Date: Mon, 13 Dec 2021 01:21:40 -0500 Subject: [PATCH 087/446] Pad: Simplify `setText` --- src/node/db/Pad.js | 18 ++++-------------- 1 file changed, 4 insertions(+), 14 deletions(-) diff --git a/src/node/db/Pad.js b/src/node/db/Pad.js index abca75102..45f8ad6af 100644 --- a/src/node/db/Pad.js +++ b/src/node/db/Pad.js @@ -256,21 +256,11 @@ Pad.prototype.text = function () { }; Pad.prototype.setText = async function (newText) { - // clean the new text newText = exports.cleanText(newText); - - const oldText = this.text(); - - // create the changeset - // We want to ensure the pad still ends with a \n, but otherwise keep - // getText() and setText() consistent. - let changeset; - if (newText[newText.length - 1] === '\n') { - changeset = Changeset.makeSplice(oldText, 0, oldText.length, newText); - } else { - changeset = Changeset.makeSplice(oldText, 0, oldText.length - 1, newText); - } - + if (!newText.endsWith('\n')) newText += '\n'; + const orig = this.text(); + if (newText === orig) return; + const changeset = Changeset.makeSplice(orig, 0, orig.length, newText); await this.appendRevision(changeset); }; From b1d0848701e358d1a9f2b99314a0bcd4478a9b63 Mon Sep 17 00:00:00 2001 From: Richard Hansen Date: Mon, 13 Dec 2021 01:29:00 -0500 Subject: [PATCH 088/446] Pad: Improve readability of `appendText` --- src/node/db/Pad.js | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/src/node/db/Pad.js b/src/node/db/Pad.js index 45f8ad6af..b7bbdb867 100644 --- a/src/node/db/Pad.js +++ b/src/node/db/Pad.js @@ -264,16 +264,16 @@ Pad.prototype.setText = async function (newText) { await this.appendRevision(changeset); }; +/** + * Appends text to the pad. + * + * @param {string} newText - Text to insert just BEFORE the pad's existing terminating newline. + */ Pad.prototype.appendText = async function (newText) { - // clean the new text newText = exports.cleanText(newText); - - const oldText = this.text(); - - // create the changeset - const changeset = Changeset.makeSplice(oldText, oldText.length, 0, newText); - - // append the changeset + const orig = this.text(); + assert(orig.endsWith('\n')); + const changeset = Changeset.makeSplice(orig, orig.length - 1, 0, newText); await this.appendRevision(changeset); }; From fdf1fdbc23c15fa623e42663014f21555edbab05 Mon Sep 17 00:00:00 2001 From: Richard Hansen Date: Sun, 12 Dec 2021 19:36:08 -0500 Subject: [PATCH 089/446] Changeset: Improve readability of `makeSplice()` --- src/static/js/Changeset.js | 36 ++++++++++++++---------------------- 1 file changed, 14 insertions(+), 22 deletions(-) diff --git a/src/static/js/Changeset.js b/src/static/js/Changeset.js index ce8e8a789..f753c7919 100644 --- a/src/static/js/Changeset.js +++ b/src/static/js/Changeset.js @@ -1482,35 +1482,27 @@ exports.identity = (N) => exports.pack(N, N, '', ''); * spliceStart+numRemoved and inserts newText instead. Also gives possibility to add attributes * optNewTextAPairs for the new text. * - * @param {string} oldFullText - old text - * @param {number} spliceStart - where splicing starts - * @param {number} numRemoved - number of characters to remove - * @param {string} newText - string to insert - * @param {string} optNewTextAPairs - new pairs to insert - * @param {AttributePool} pool - Attribute pool + * @param {string} orig - Original text. + * @param {number} start - Index into `orig` where characters should be removed and inserted. + * @param {number} ndel - Number of characters to delete at `start`. + * @param {string} ins - Text to insert at `start` (after deleting `ndel` characters). + * @param {string} [attribs] - Optional attributes to apply to the inserted text. + * @param {AttributePool} [pool] - Attribute pool. * @returns {string} */ -exports.makeSplice = (oldFullText, spliceStart, numRemoved, newText, optNewTextAPairs, pool) => { - const oldLen = oldFullText.length; - - if (spliceStart >= oldLen) { - spliceStart = oldLen - 1; - } - if (numRemoved > oldFullText.length - spliceStart) { - numRemoved = oldFullText.length - spliceStart; - } - const oldText = oldFullText.substring(spliceStart, spliceStart + numRemoved); - const newLen = oldLen + newText.length - oldText.length; - +exports.makeSplice = (orig, start, ndel, ins, attribs, pool) => { + if (start >= orig.length) start = orig.length - 1; + if (ndel > orig.length - start) ndel = orig.length - start; + const deleted = orig.substring(start, start + ndel); const assem = exports.smartOpAssembler(); const ops = (function* () { - yield* opsFromText('=', oldFullText.substring(0, spliceStart)); - yield* opsFromText('-', oldText); - yield* opsFromText('+', newText, optNewTextAPairs, pool); + yield* opsFromText('=', orig.substring(0, start)); + yield* opsFromText('-', deleted); + yield* opsFromText('+', ins, attribs, pool); })(); for (const op of ops) assem.append(op); assem.endDocument(); - return exports.pack(oldLen, newLen, assem.toString(), newText); + return exports.pack(orig.length, orig.length + ins.length - ndel, assem.toString(), ins); }; /** From 30d68df3966c544ba23f4edb63785d17252968cf Mon Sep 17 00:00:00 2001 From: Richard Hansen Date: Sun, 12 Dec 2021 23:50:48 -0500 Subject: [PATCH 090/446] Changeset: Add range checks to `makeSplice` --- src/static/js/Changeset.js | 2 ++ 1 file changed, 2 insertions(+) diff --git a/src/static/js/Changeset.js b/src/static/js/Changeset.js index f753c7919..81032b580 100644 --- a/src/static/js/Changeset.js +++ b/src/static/js/Changeset.js @@ -1491,6 +1491,8 @@ exports.identity = (N) => exports.pack(N, N, '', ''); * @returns {string} */ exports.makeSplice = (orig, start, ndel, ins, attribs, pool) => { + if (start < 0) throw new RangeError(`start index must be non-negative (is ${start})`); + if (ndel < 0) throw new RangeError(`characters to delete must be non-negative (is ${ndel})`); if (start >= orig.length) start = orig.length - 1; if (ndel > orig.length - start) ndel = orig.length - start; const deleted = orig.substring(start, start + ndel); From 748d6614951312d52abc7d493e07acaacbbaa1b6 Mon Sep 17 00:00:00 2001 From: Richard Hansen Date: Mon, 13 Dec 2021 00:04:05 -0500 Subject: [PATCH 091/446] Changeset: Fix off-by-one bug in `makeSplice` --- src/static/js/Changeset.js | 2 +- src/tests/frontend/specs/easysync-other.js | 7 +++++++ 2 files changed, 8 insertions(+), 1 deletion(-) diff --git a/src/static/js/Changeset.js b/src/static/js/Changeset.js index 81032b580..e3ae9d286 100644 --- a/src/static/js/Changeset.js +++ b/src/static/js/Changeset.js @@ -1493,7 +1493,7 @@ exports.identity = (N) => exports.pack(N, N, '', ''); exports.makeSplice = (orig, start, ndel, ins, attribs, pool) => { if (start < 0) throw new RangeError(`start index must be non-negative (is ${start})`); if (ndel < 0) throw new RangeError(`characters to delete must be non-negative (is ${ndel})`); - if (start >= orig.length) start = orig.length - 1; + if (start > orig.length) start = orig.length; if (ndel > orig.length - start) ndel = orig.length - start; const deleted = orig.substring(start, start + ndel); const assem = exports.smartOpAssembler(); diff --git a/src/tests/frontend/specs/easysync-other.js b/src/tests/frontend/specs/easysync-other.js index 856b7df62..af4580835 100644 --- a/src/tests/frontend/specs/easysync-other.js +++ b/src/tests/frontend/specs/easysync-other.js @@ -70,6 +70,13 @@ describe('easysync-other', function () { expect(t2).to.equal('a\nb\ncdef\n'); }); + it('makeSplice at the end', async function () { + const orig = '123'; + const ins = '456'; + expect(Changeset.applyToText(Changeset.makeSplice(orig, orig.length, 0, ins), orig)) + .to.equal(`${orig}${ins}`); + }); + it('testToSplices', async function () { const cs = Changeset.checkRep('Z:z>9*0=1=4-3+9=1|1-4-4+1*0+a$123456789abcdefghijk'); const correctSplices = [ From e64462323bc69219f4fbc78267a84730b72137a9 Mon Sep 17 00:00:00 2001 From: "translatewiki.net" Date: Thu, 16 Dec 2021 13:03:26 +0100 Subject: [PATCH 092/446] Localisation updates from https://translatewiki.net. --- src/locales/zh-hans.json | 33 +++++++++++++++++++++++++++++++++ 1 file changed, 33 insertions(+) diff --git a/src/locales/zh-hans.json b/src/locales/zh-hans.json index e2dca09a8..5ea5939e1 100644 --- a/src/locales/zh-hans.json +++ b/src/locales/zh-hans.json @@ -11,6 +11,7 @@ "Qiyue2001", "Shangkuanlc", "Shizhao", + "Stang", "VulpesVulpes825", "Yfdyh000", "乌拉跨氪", @@ -19,6 +20,36 @@ "燃玉" ] }, + "admin.page-title": "管理员面板 - Etherpad", + "admin_plugins": "插件管理器", + "admin_plugins.available": "可用插件", + "admin_plugins.available_not-found": "找不到插件。", + "admin_plugins.available_fetching": "获取中…", + "admin_plugins.available_install.value": "安装", + "admin_plugins.available_search.placeholder": "搜索插件以安装", + "admin_plugins.description": "描述", + "admin_plugins.installed": "已装插件", + "admin_plugins.installed_fetching": "正在获取已安装的插件…", + "admin_plugins.installed_nothing": "您尚未安装任何插件。", + "admin_plugins.installed_uninstall.value": "卸载", + "admin_plugins.last-update": "最后更新", + "admin_plugins.name": "名称", + "admin_plugins.page-title": "插件管理器 - Etherpad", + "admin_plugins.version": "版本", + "admin_plugins_info": "故障排除信息", + "admin_plugins_info.parts": "已安装部分", + "admin_plugins_info.plugins": "已安装插件", + "admin_plugins_info.page-title": "插件信息 - Etherpad", + "admin_plugins_info.version": "Etherpad版本", + "admin_plugins_info.version_latest": "最新可用版本", + "admin_plugins_info.version_number": "版本号", + "admin_settings": "设置", + "admin_settings.current": "当前配置", + "admin_settings.current_example-devel": "开发设置模板示例", + "admin_settings.current_example-prod": "生产设置模板示例", + "admin_settings.current_restart.value": "重启Etherpad", + "admin_settings.current_save.value": "保存设置", + "admin_settings.page-title": "设置 - Etherpad", "index.newPad": "新记事本", "index.createOpenPad": "或者创建/打开带名字的记事本：", "index.openPad": "打开一个现有的记事本，名称为：", @@ -92,6 +123,8 @@ "pad.modals.deleted.explanation": "此记事本已被移除。", "pad.modals.rateLimited": "速率限制", "pad.modals.rateLimited.explanation": "您向此记事本发送了太多消息，因此中断了与您的连接。", + "pad.modals.rejected.explanation": "服务器拒绝了您的浏览器发送的信息。", + "pad.modals.rejected.cause": "服务器可能在你查看页面时更新了，也可能是Etherpad出现了错误。请尝试重新加载页面。", "pad.modals.disconnected": "您已断开连接。", "pad.modals.disconnected.explanation": "到服务器的连接已丢失", "pad.modals.disconnected.cause": "服务器可能无法使用。若此情况持续发生，请通知服务器管理员。", From 0040f5984e4052e7b8cd2b503b9f14b72be1fa29 Mon Sep 17 00:00:00 2001 From: webzwo0i Date: Sun, 4 Jul 2021 16:30:30 +0200 Subject: [PATCH 093/446] db: await more database operations Co-authored-by: Richard Hansen --- src/node/db/API.js | 4 ++-- src/node/db/Pad.js | 28 ++++++++++++---------------- src/node/db/ReadOnlyManager.js | 8 +++++--- 3 files changed, 19 insertions(+), 21 deletions(-) diff --git a/src/node/db/API.js b/src/node/db/API.js index 040abf5a6..7fe0e5ca1 100644 --- a/src/node/db/API.js +++ b/src/node/db/API.js @@ -139,11 +139,11 @@ exports.getRevisionChangeset = async (padID, rev) => { } // get the changeset for this revision - return pad.getRevisionChangeset(rev); + return await pad.getRevisionChangeset(rev); } // the client wants the latest changeset, lets return it to him - return pad.getRevisionChangeset(head); + return await pad.getRevisionChangeset(head); }; /** diff --git a/src/node/db/Pad.js b/src/node/db/Pad.js index b7bbdb867..cca065dfc 100644 --- a/src/node/db/Pad.js +++ b/src/node/db/Pad.js @@ -376,12 +376,12 @@ Pad.prototype.copy = async function (destinationID, force) { // if force is true and already exists a Pad with the same id, remove that Pad await this.removePadIfForceIsTrueAndAlreadyExist(destinationID, force); - // copy the 'pad' entry - const pad = await this._db.get(`pad:${this.id}`); - db.set(`pad:${destinationID}`, pad); - - // copy all relations in parallel - const promises = []; + // copy all records in parallel + const promises = [ + // Copy the 'pad' entry. This is wrapped in an IIFE so that this._db.get() can run in parallel + // with the other record copies done below. + (async () => await db.set(`pad:${destinationID}`, await this._db.get(`pad:${this.id}`)))(), + ]; // copy all chat messages const chatHead = this.chatHead; @@ -399,7 +399,7 @@ Pad.prototype.copy = async function (destinationID, force) { promises.push(p); } - this.copyAuthorInfoToDestinationPad(destinationID); + promises.push(this.copyAuthorInfoToDestinationPad(destinationID)); // wait for the above to complete await Promise.all(promises); @@ -409,11 +409,8 @@ Pad.prototype.copy = async function (destinationID, force) { await db.setSub(`group:${destGroupID}`, ['pads', destinationID], 1); } - // delay still necessary? - await new Promise((resolve) => setTimeout(resolve, 10)); - // Initialize the new pad (will update the listAllPads cache) - await padManager.getPad(destinationID, null); // this runs too early. + await padManager.getPad(destinationID, null); // let the plugins know the pad was copied await hooks.aCallAll('padCopy', {originalPad: this, destinationID}); @@ -459,11 +456,10 @@ Pad.prototype.removePadIfForceIsTrueAndAlreadyExist = async function (destinatio } }; -Pad.prototype.copyAuthorInfoToDestinationPad = function (destinationID) { +Pad.prototype.copyAuthorInfoToDestinationPad = async function (destinationID) { // add the new sourcePad to all authors who contributed to the old one - this.getAllAuthors().forEach((authorID) => { - authorManager.addPad(authorID, destinationID); - }); + await Promise.all(this.getAllAuthors().map( + (authorID) => authorManager.addPad(authorID, destinationID))); }; Pad.prototype.copyPadWithoutHistory = async function (destinationID, force) { @@ -481,7 +477,7 @@ Pad.prototype.copyPadWithoutHistory = async function (destinationID, force) { const sourcePad = await padManager.getPad(sourceID); // add the new sourcePad to all authors who contributed to the old one - this.copyAuthorInfoToDestinationPad(destinationID); + await this.copyAuthorInfoToDestinationPad(destinationID); // Group pad? Add it to the group's list if (destGroupID) { diff --git a/src/node/db/ReadOnlyManager.js b/src/node/db/ReadOnlyManager.js index 0b8d29171..33ce2930a 100644 --- a/src/node/db/ReadOnlyManager.js +++ b/src/node/db/ReadOnlyManager.js @@ -41,8 +41,10 @@ exports.getReadOnlyId = async (padId) => { // there is no readOnly Entry in the database, let's create one if (readOnlyId == null) { readOnlyId = `r.${randomString(16)}`; - db.set(`pad2readonly:${padId}`, readOnlyId); - db.set(`readonly2pad:${readOnlyId}`, padId); + await Promise.all([ + db.set(`pad2readonly:${padId}`, readOnlyId), + db.set(`readonly2pad:${readOnlyId}`, padId), + ]); } return readOnlyId; @@ -52,7 +54,7 @@ exports.getReadOnlyId = async (padId) => { * returns the padId for a read only id * @param {String} readOnlyId read only id */ -exports.getPadId = (readOnlyId) => db.get(`readonly2pad:${readOnlyId}`); +exports.getPadId = async (readOnlyId) => await db.get(`readonly2pad:${readOnlyId}`); /** * returns the padId and readonlyPadId in an object for any id From 757204083620b60d0f7c118609b517928b3cc405 Mon Sep 17 00:00:00 2001 From: Richard Hansen Date: Fri, 17 Sep 2021 00:30:25 -0400 Subject: [PATCH 094/446] Pad: Simplify `Pad.copy()` logic --- src/node/db/Pad.js | 42 +++++++++++------------------------------- 1 file changed, 11 insertions(+), 31 deletions(-) diff --git a/src/node/db/Pad.js b/src/node/db/Pad.js index cca065dfc..cba2f2d74 100644 --- a/src/node/db/Pad.js +++ b/src/node/db/Pad.js @@ -376,38 +376,18 @@ Pad.prototype.copy = async function (destinationID, force) { // if force is true and already exists a Pad with the same id, remove that Pad await this.removePadIfForceIsTrueAndAlreadyExist(destinationID, force); - // copy all records in parallel - const promises = [ - // Copy the 'pad' entry. This is wrapped in an IIFE so that this._db.get() can run in parallel - // with the other record copies done below. - (async () => await db.set(`pad:${destinationID}`, await this._db.get(`pad:${this.id}`)))(), - ]; + const copyRecord = async (keySuffix) => { + const val = await this._db.get(`pad:${this.id}${keySuffix}`); + await db.set(`pad:${destinationID}${keySuffix}`, val); + }; - // copy all chat messages - const chatHead = this.chatHead; - for (let i = 0; i <= chatHead; ++i) { - const p = this._db.get(`pad:${this.id}:chat:${i}`) - .then((chat) => db.set(`pad:${destinationID}:chat:${i}`, chat)); - promises.push(p); - } - - // copy all revisions - const revHead = this.head; - for (let i = 0; i <= revHead; ++i) { - const p = this._db.get(`pad:${this.id}:revs:${i}`) - .then((rev) => db.set(`pad:${destinationID}:revs:${i}`, rev)); - promises.push(p); - } - - promises.push(this.copyAuthorInfoToDestinationPad(destinationID)); - - // wait for the above to complete - await Promise.all(promises); - - // Group pad? Add it to the group's list - if (destGroupID) { - await db.setSub(`group:${destGroupID}`, ['pads', destinationID], 1); - } + await Promise.all((function* () { + yield copyRecord(''); + for (let i = 0; i <= this.head; ++i) yield copyRecord(`:revs:${i}`); + for (let i = 0; i <= this.chatHead; ++i) yield copyRecord(`:chat:${i}`); + yield this.copyAuthorInfoToDestinationPad(destinationID); + if (destGroupID) yield db.setSub(`group:${destGroupID}`, ['pads', destinationID], 1); + }).call(this)); // Initialize the new pad (will update the listAllPads cache) await padManager.getPad(destinationID, null); From 694d3f630edb286ba5b1169906adf2e5edc986de Mon Sep 17 00:00:00 2001 From: webzwo0i Date: Mon, 5 Jul 2021 06:07:40 +0200 Subject: [PATCH 095/446] SessionStore: Propagate database errors to express-session Send a 500 HTTP status code to the client if the session entry could not be fetched from the database. This is useful in case the database is busy and can't respond to the query in time. In this case we want to abort the client connection as soon as possible. Co-authored-by: Richard Hansen --- src/node/db/SessionStore.js | 18 ++++++------------ 1 file changed, 6 insertions(+), 12 deletions(-) diff --git a/src/node/db/SessionStore.js b/src/node/db/SessionStore.js index 2c5d1ca25..d06754fdd 100644 --- a/src/node/db/SessionStore.js +++ b/src/node/db/SessionStore.js @@ -17,18 +17,12 @@ const logger = log4js.getLogger('SessionStore'); module.exports = class SessionStore extends Store { get(sid, fn) { logger.debug(`GET ${sid}`); - DB.db.get(`sessionstorage:${sid}`, (err, sess) => { - if (sess) { - sess.cookie.expires = ('string' === typeof sess.cookie.expires - ? new Date(sess.cookie.expires) : sess.cookie.expires); - if (!sess.cookie.expires || new Date() < sess.cookie.expires) { - fn(null, sess); - } else { - this.destroy(sid, fn); - } - } else { - fn(); - } + DB.db.get(`sessionstorage:${sid}`, (err, s) => { + if (err != null) return fn(err); + if (!s) return fn(null); + if (typeof s.cookie.expires === 'string') s.cookie.expires = new Date(s.cookie.expires); + if (s.cookie.expires && new Date() >= s.cookie.expires) return this.destroy(sid, fn); + fn(null, s); }); } From 4733c7d8d325e478359bc22b9313ba29eb598ad2 Mon Sep 17 00:00:00 2001 From: Richard Hansen Date: Thu, 16 Sep 2021 23:01:10 -0400 Subject: [PATCH 096/446] SessionStore: Promisify to the extent permitted by express-session --- src/node/db/SessionStore.js | 46 +++++++++++++++++++------------------ 1 file changed, 24 insertions(+), 22 deletions(-) diff --git a/src/node/db/SessionStore.js b/src/node/db/SessionStore.js index d06754fdd..f4b26d4be 100644 --- a/src/node/db/SessionStore.js +++ b/src/node/db/SessionStore.js @@ -1,38 +1,40 @@ 'use strict'; -/* - * Stores session data in the database - * Source; https://github.com/edy-b/SciFlowWriter/blob/develop/available_plugins/ep_sciflowwriter/db/DirtyStore.js - * This is not used for authors that are created via the API at current - * - * RPB: this module was not migrated to Promises, because it is only used via - * express-session, which can't actually use promises anyway. - */ const DB = require('./DB'); const Store = require('express-session').Store; const log4js = require('log4js'); +const util = require('util'); const logger = log4js.getLogger('SessionStore'); -module.exports = class SessionStore extends Store { - get(sid, fn) { +class SessionStore extends Store { + async _get(sid) { logger.debug(`GET ${sid}`); - DB.db.get(`sessionstorage:${sid}`, (err, s) => { - if (err != null) return fn(err); - if (!s) return fn(null); - if (typeof s.cookie.expires === 'string') s.cookie.expires = new Date(s.cookie.expires); - if (s.cookie.expires && new Date() >= s.cookie.expires) return this.destroy(sid, fn); - fn(null, s); - }); + const s = await DB.get(`sessionstorage:${sid}`); + if (!s) return; + if (typeof s.cookie.expires === 'string') s.cookie.expires = new Date(s.cookie.expires); + if (s.cookie.expires && new Date() >= s.cookie.expires) { + await this._destroy(sid); + return; + } + return s; } - set(sid, sess, fn) { + async _set(sid, sess) { logger.debug(`SET ${sid}`); - DB.db.set(`sessionstorage:${sid}`, sess, fn); + await DB.set(`sessionstorage:${sid}`, sess); } - destroy(sid, fn) { + async _destroy(sid) { logger.debug(`DESTROY ${sid}`); - DB.db.remove(`sessionstorage:${sid}`, fn); + await DB.remove(`sessionstorage:${sid}`); } -}; +} + +// express-session doesn't support Promise-based methods. This is where the callbackified versions +// used by express-session are defined. +for (const m of ['get', 'set', 'destroy']) { + SessionStore.prototype[m] = util.callbackify(SessionStore.prototype[`_${m}`]); +} + +module.exports = SessionStore; From 8b73f2ee70d79342d37a066c7773430773d47cf4 Mon Sep 17 00:00:00 2001 From: webzwo0i Date: Mon, 5 Jul 2021 06:12:02 +0200 Subject: [PATCH 097/446] padurlsanitize: Don't crash if `sanitizePadId()` throws Let Express send a 500 status code to the user instead. Co-authored-by: Richard Hansen --- CHANGELOG.md | 1 + src/node/hooks/express/padurlsanitize.js | 35 +++++++++++++----------- 2 files changed, 20 insertions(+), 16 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 14653a33c..488d0e947 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -15,6 +15,7 @@ from the database when the group is deleted. * Fixed race conditions in the `setText`, `appendText`, and `restoreRevision` functions (HTTP API). +* Fixed a crash if the database is busy enough to cause a query timeout. #### For plugin authors diff --git a/src/node/hooks/express/padurlsanitize.js b/src/node/hooks/express/padurlsanitize.js index b805fc4ba..ff1afa477 100644 --- a/src/node/hooks/express/padurlsanitize.js +++ b/src/node/hooks/express/padurlsanitize.js @@ -4,24 +4,27 @@ const padManager = require('../../db/PadManager'); exports.expressCreateServer = (hookName, args, cb) => { // redirects browser to the pad's sanitized url if needed. otherwise, renders the html - args.app.param('pad', async (req, res, next, padId) => { - // ensure the padname is valid and the url doesn't end with a / - if (!padManager.isValidPadId(padId) || /\/$/.test(req.url)) { - res.status(404).send('Such a padname is forbidden'); - return; - } + args.app.param('pad', (req, res, next, padId) => { + (async () => { + // ensure the padname is valid and the url doesn't end with a / + if (!padManager.isValidPadId(padId) || /\/$/.test(req.url)) { + res.status(404).send('Such a padname is forbidden'); + return; + } - const sanitizedPadId = await padManager.sanitizePadId(padId); + const sanitizedPadId = await padManager.sanitizePadId(padId); - if (sanitizedPadId === padId) { - // the pad id was fine, so just render it - next(); - } else { - // the pad id was sanitized, so we redirect to the sanitized version - const realURL = encodeURIComponent(sanitizedPadId) + new URL(req.url, 'http://invalid.invalid').search; - res.header('Location', realURL); - res.status(302).send(`You should be redirected to ${realURL}`); - } + if (sanitizedPadId === padId) { + // the pad id was fine, so just render it + next(); + } else { + // the pad id was sanitized, so we redirect to the sanitized version + const realURL = + encodeURIComponent(sanitizedPadId) + new URL(req.url, 'http://invalid.invalid').search; + res.header('Location', realURL); + res.status(302).send(`You should be redirected to ${realURL}`); + } + })().catch((err) => next(err || new Error(err))); }); return cb(); }; From 674a0ccedc47dd1a325c2be3435e39bedde92f2a Mon Sep 17 00:00:00 2001 From: snyk-bot Date: Sun, 19 Dec 2021 00:30:20 +0000 Subject: [PATCH 098/446] fix: upgrade openapi-backend from 5.0.0 to 5.0.1 Snyk has created this PR to upgrade openapi-backend from 5.0.0 to 5.0.1. See this package in npm: https://www.npmjs.com/package/openapi-backend See this project in Snyk: https://app.snyk.io/org/johnmclear/project/d9a12bfb-7ccd-443f-9e22-f30d339cc8c5?utm_source=github&utm_medium=referral&page=upgrade-pr --- src/package-lock.json | 12 ++++++------ src/package.json | 2 +- 2 files changed, 7 insertions(+), 7 deletions(-) diff --git a/src/package-lock.json b/src/package-lock.json index baefcb422..22304dde4 100644 --- a/src/package-lock.json +++ b/src/package-lock.json @@ -7330,9 +7330,9 @@ } }, "openapi-backend": { - "version": "5.0.0", - "resolved": "https://registry.npmjs.org/openapi-backend/-/openapi-backend-5.0.0.tgz", - "integrity": "sha512-ppOFSLEXIgmRGcbd9kfPzXmUFHhlBux69rlu7dot5XZ6BH7ycXEvFjRdFLXZ76GdO++i3epDZkxkRBHqPNoz/Q==", + "version": "5.0.1", + "resolved": "https://registry.npmjs.org/openapi-backend/-/openapi-backend-5.0.1.tgz", + "integrity": "sha512-DHjQ7d3izTgIy6lSD+WGLhlunZ+fkyleHjNnUSGjHC7Lii3ukuB3N/J74NaCHf5mrDlXgCfylFMRVtq4oDwPKQ==", "requires": { "@apidevtools/json-schema-ref-parser": "^9.0.7", "ajv": "^8.6.2", @@ -7346,9 +7346,9 @@ }, "dependencies": { "qs": { - "version": "6.10.1", - "resolved": "https://registry.npmjs.org/qs/-/qs-6.10.1.tgz", - "integrity": "sha512-M528Hph6wsSVOBiYUnGf+K/7w0hNshs/duGsNXPUCLH5XAqjEtiPGwNONLV0tBH8NoGb0mvD5JubnUTrujKDTg==", + "version": "6.10.2", + "resolved": "https://registry.npmjs.org/qs/-/qs-6.10.2.tgz", + "integrity": "sha512-mSIdjzqznWgfd4pMii7sHtaYF8rx8861hBO80SraY5GT0XQibWZWJSid0avzHGkDIZLImux2S5mXO0Hfct2QCw==", "requires": { "side-channel": "^1.0.4" } diff --git a/src/package.json b/src/package.json index 4daac1c46..db8f5ad3c 100644 --- a/src/package.json +++ b/src/package.json @@ -53,7 +53,7 @@ "measured-core": "^2.0.0", "mime-types": "^2.1.34", "npm": "^6.14.15", - "openapi-backend": "^5.0.0", + "openapi-backend": "^5.0.1", "proxy-addr": "^2.0.7", "rate-limiter-flexible": "^2.3.5", "rehype": "^11.0.0", From 02d1b90d30c05109bcc3d086e9ca5a927d2432f2 Mon Sep 17 00:00:00 2001 From: Richard Hansen Date: Sun, 19 Dec 2021 16:47:45 -0500 Subject: [PATCH 099/446] tests: Factor out USER_CHANGES/ACCEPT_COMMIT helpers This will make it possible for other tests to reuse the code. --- src/tests/backend/common.js | 54 ++++++++++++++++++++++ src/tests/backend/specs/messages.js | 72 ++++++++++++++--------------- 2 files changed, 90 insertions(+), 36 deletions(-) diff --git a/src/tests/backend/common.js b/src/tests/backend/common.js index 89f635012..1787ace7d 100644 --- a/src/tests/backend/common.js +++ b/src/tests/backend/common.js @@ -1,6 +1,8 @@ 'use strict'; +const AttributePool = require('../../static/js/AttributePool'); const apiHandler = require('../../node/handler/APIHandler'); +const assert = require('assert').strict; const io = require('socket.io-client'); const log4js = require('log4js'); const process = require('process'); @@ -185,6 +187,58 @@ exports.handshake = async (socket, padId) => { return msg; }; +/** + * Convenience wrapper around `socket.send()` that waits for acknowledgement. + */ +exports.sendMessage = async (socket, message) => await new Promise((resolve, reject) => { + socket.send(message, (errInfo) => { + if (errInfo != null) { + const {name, message} = errInfo; + const err = new Error(message); + err.name = name; + reject(err); + return; + } + resolve(); + }); +}); + +/** + * Convenience function to send a USER_CHANGES message. Waits for acknowledgement. + */ +exports.sendUserChanges = async (socket, data) => await exports.sendMessage(socket, { + type: 'COLLABROOM', + component: 'pad', + data: { + type: 'USER_CHANGES', + apool: new AttributePool(), + ...data, + }, +}); + +/** + * Convenience function that waits for an ACCEPT_COMMIT message. Asserts that the new revision + * matches the expected revision. + * + * Note: To avoid a race condition, this should be called before the USER_CHANGES message is sent. + * For example: + * + * await Promise.all([ + * common.waitForAcceptCommit(socket, rev + 1), + * common.sendUserChanges(socket, {baseRev: rev, changeset}), + * ]); + */ +exports.waitForAcceptCommit = async (socket, wantRev) => { + const msg = await exports.waitForSocketEvent(socket, 'message'); + assert.deepEqual(msg, { + type: 'COLLABROOM', + data: { + type: 'ACCEPT_COMMIT', + newRev: wantRev, + }, + }); +}; + const alphabet = 'abcdefghijklmnopqrstuvwxyz'; /** diff --git a/src/tests/backend/specs/messages.js b/src/tests/backend/specs/messages.js index 4c9f7e66c..2d5546d6c 100644 --- a/src/tests/backend/specs/messages.js +++ b/src/tests/backend/specs/messages.js @@ -36,27 +36,10 @@ describe(__filename, function () { }); describe('USER_CHANGES', function () { - const sendUserChanges = (changeset, apool = new AttributePool()) => { - socket.json.send({ - type: 'COLLABROOM', - component: 'pad', - data: { - type: 'USER_CHANGES', - baseRev: rev, - changeset, - apool: new AttributePool(), - }, - }); - }; + const sendUserChanges = + async (changeset) => await common.sendUserChanges(socket, {baseRev: rev, changeset}); const assertAccepted = async (wantRev) => { - const msg = await common.waitForSocketEvent(socket, 'message'); - assert.deepEqual(msg, { - type: 'COLLABROOM', - data: { - type: 'ACCEPT_COMMIT', - newRev: wantRev, - }, - }); + await common.waitForAcceptCommit(socket, wantRev); rev = wantRev; }; const assertRejected = async () => { @@ -65,38 +48,55 @@ describe(__filename, function () { }; it('changes are applied', async function () { - sendUserChanges('Z:1>5+5$hello'); - await assertAccepted(rev + 1); + await Promise.all([ + assertAccepted(rev + 1), + sendUserChanges('Z:1>5+5$hello'), + ]); assert.equal(pad.text(), 'hello\n'); }); it('bad changeset is rejected', async function () { - sendUserChanges('this is not a valid changeset'); - await assertRejected(); + await Promise.all([ + assertRejected(), + sendUserChanges('this is not a valid changeset'), + ]); }); it('retransmission is accepted, has no effect', async function () { - sendUserChanges('Z:1>5+5$hello'); - await assertAccepted(rev + 1); + const cs = 'Z:1>5+5$hello'; + await Promise.all([ + assertAccepted(rev + 1), + sendUserChanges(cs), + ]); --rev; - sendUserChanges('Z:1>5+5$hello'); - await assertAccepted(rev + 1); + await Promise.all([ + assertAccepted(rev + 1), + sendUserChanges(cs), + ]); assert.equal(pad.text(), 'hello\n'); }); it('identity changeset is accepted, has no effect', async function () { - sendUserChanges('Z:1>5+5$hello'); - await assertAccepted(rev + 1); - sendUserChanges('Z:6>0$'); - await assertAccepted(rev); + await Promise.all([ + assertAccepted(rev + 1), + sendUserChanges('Z:1>5+5$hello'), + ]); + await Promise.all([ + assertAccepted(rev), + sendUserChanges('Z:6>0$'), + ]); assert.equal(pad.text(), 'hello\n'); }); it('non-identity changeset with no net change is accepted, has no effect', async function () { - sendUserChanges('Z:1>5+5$hello'); - await assertAccepted(rev + 1); - sendUserChanges('Z:6>0-5+5$hello'); - await assertAccepted(rev); + await Promise.all([ + assertAccepted(rev + 1), + sendUserChanges('Z:1>5+5$hello'), + ]); + await Promise.all([ + assertAccepted(rev), + sendUserChanges('Z:6>0-5+5$hello'), + ]); assert.equal(pad.text(), 'hello\n'); }); }); From c4b25388ae3460efafa11ca51fa053f377c158a1 Mon Sep 17 00:00:00 2001 From: Richard Hansen Date: Fri, 17 Dec 2021 16:27:17 -0500 Subject: [PATCH 100/446] docs: Server-side hook documentation improvements --- doc/api/hooks_server-side.md | 91 +++++++++++++++++++++--------------- 1 file changed, 53 insertions(+), 38 deletions(-) diff --git a/doc/api/hooks_server-side.md b/doc/api/hooks_server-side.md index f7aa6b0b6..ddae10d14 100644 --- a/doc/api/hooks_server-side.md +++ b/doc/api/hooks_server-side.md @@ -58,24 +58,34 @@ Run during startup after the named plugin is initialized. Context properties: None -## expressConfigure -Called from: src/node/hooks/express.js +## `expressConfigure` -Things in context: +Called from: `src/node/hooks/express.js` -1. app - the main application object +Called during server startup just after the +[`express-session`](https://www.npmjs.com/package/express-session) middleware is +added to the Express Application object. Use this hook to add route handlers or +middleware that executes after `express-session` state is created and +authentication is performed. -This is a helpful hook for changing the behavior and configuration of the application. It's called right after the application gets configured. +Context properties: -## expressCreateServer -Called from: src/node/hooks/express.js +* `app`: The Express [Application](https://expressjs.com/en/4x/api.html#app) + object. -Things in context: +## `expressCreateServer` -1. app - the main express application object (helpful for adding new paths and such) -2. server - the http server object +Called from: `src/node/hooks/express.js` -This hook gets called after the application object has been created, but before it starts listening. This is similar to the expressConfigure hook, but it's not guaranteed that the application object will have all relevant configuration variables. +Identical to the `expressConfigure` hook (the two run in parallel with each +other) except this hook's context includes the HTTP Server object. + +Context properties: + +* `app`: The Express [Application](https://expressjs.com/en/4x/api.html#app) + object. +* `server`: The [http.Server](https://nodejs.org/api/http.html#class-httpserver) + or [https.Server](https://nodejs.org/api/https.html#class-httpsserver) object. ## expressCloseServer @@ -235,47 +245,52 @@ Things in context: I have no idea what this is useful for, someone else will have to add this description. -## preAuthorize -Called from: src/node/hooks/express/webaccess.js +## `preAuthorize` -Things in context: +Called from: `src/node/hooks/express/webaccess.js` -1. req - the request object -2. res - the response object -3. next - bypass callback. If this is called instead of the normal callback then - all remaining access checks are skipped. +Called for each HTTP request before any authentication checks are performed. The +registered `preAuthorize` hook functions are called one at a time until one +explicitly grants or denies the request by returning `true` or `false`, +respectively. If none of the hook functions return anything, the access decision +is deferred to the normal authentication and authorization checks. -This hook is called for each HTTP request before any authentication checks are -performed. Example uses: +Example uses: * Always grant access to static content. * Process an OAuth callback. * Drop requests from IP addresses that have failed N authentication checks within the past X minutes. -A preAuthorize function is always called for each request unless a preAuthorize -function from another plugin (if any) has already explicitly granted or denied -the request. +Return values: -You can pass the following values to the provided callback: +* `undefined` (or `[]`) defers the access decision to the next registered + `preAuthorize` hook function, or to the normal authentication and + authorization checks if no more `preAuthorize` hook functions remain. +* `true` (or `[true]`) immediately grants access to the requested resource, + unless the request is for an `/admin` page in which case it is treated the + same as returning `undefined`. (This prevents buggy plugins from accidentally + granting admin access to the general public.) +* `false` (or `[false]`) immediately denies the request. The `preAuthnFailure` + hook will be called to handle the failure. -* `[]` defers the access decision to the normal authentication and authorization - checks (or to a preAuthorize function from another plugin, if one exists). -* `[true]` immediately grants access to the requested resource, unless the - request is for an `/admin` page in which case it is treated the same as `[]`. - (This prevents buggy plugins from accidentally granting admin access to the - general public.) -* `[false]` immediately denies the request. The preAuthnFailure hook will be - called to handle the failure. +Context properties: + +* `req`: The Express [Request](https://expressjs.com/en/4x/api.html#req) object. +* `res`: The Express [Response](https://expressjs.com/en/4x/api.html#res) + object. +* `next`: Callback to immediately hand off handling to the next Express + middleware/handler, or to the next matching route if `'route'` is passed as + the first argument. Do not call this unless you understand the consequences. Example: -``` -exports.preAuthorize = (hookName, context, cb) => { - if (ipAddressIsFirewalled(context.req)) return cb([false]); - if (requestIsForStaticContent(context.req)) return cb([true]); - if (requestIsForOAuthCallback(context.req)) return cb([true]); - return cb([]); +```javascript +exports.preAuthorize = async (hookName, {req}) => { + if (await ipAddressIsFirewalled(req)) return false; + if (requestIsForStaticContent(req)) return true; + if (requestIsForOAuthCallback(req)) return true; + // Defer the decision to the next step by returning undefined. }; ``` From fc498f0ae67c06d00ab3a797252b61e7890779c5 Mon Sep 17 00:00:00 2001 From: Richard Hansen Date: Fri, 17 Dec 2021 23:41:45 -0500 Subject: [PATCH 101/446] tests: Delete test pad before attempting import --- .../backend/specs/api/importexportGetPost.js | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-) diff --git a/src/tests/backend/specs/api/importexportGetPost.js b/src/tests/backend/specs/api/importexportGetPost.js index 98244b22b..11498384d 100644 --- a/src/tests/backend/specs/api/importexportGetPost.js +++ b/src/tests/backend/specs/api/importexportGetPost.js @@ -25,6 +25,13 @@ const apiVersion = 1; const testPadId = makeid(); const testPadIdEnc = encodeURIComponent(testPadId); +const deleteTestPad = async () => { + if (await padManager.doesPadExist(testPadId)) { + const pad = await padManager.getPad(testPadId); + await pad.remove(); + } +}; + describe(__filename, function () { this.timeout(45000); before(async function () { agent = await common.init(); }); @@ -364,6 +371,7 @@ describe(__filename, function () { // makeGoodExport() is assumed to produce good .etherpad records. Verify that assumption so // that a buggy makeGoodExport() doesn't cause checks to accidentally pass. const records = makeGoodExport(); + await deleteTestPad(); await importEtherpad(records) .expect(200) .expect('Content-Type', /json/) @@ -430,13 +438,6 @@ describe(__filename, function () { describe('Import authorization checks', function () { let authorize; - const deleteTestPad = async () => { - if (await padManager.doesPadExist(testPadId)) { - const pad = await padManager.getPad(testPadId); - await pad.remove(); - } - }; - const createTestPad = async (text) => { const pad = await padManager.getPad(testPadId); if (text) await pad.setText(text); From 472eddc821db1851e3645f5eee106f9b217af16e Mon Sep 17 00:00:00 2001 From: Richard Hansen Date: Sat, 18 Dec 2021 00:55:20 -0500 Subject: [PATCH 102/446] webaccess: Skip checks if `next` is called in `preAuthenticate` --- src/node/hooks/express/webaccess.js | 10 +++++++--- src/tests/backend/specs/webaccess.js | 15 +++++++++++---- 2 files changed, 18 insertions(+), 7 deletions(-) diff --git a/src/node/hooks/express/webaccess.js b/src/node/hooks/express/webaccess.js index 8a183681c..3d47b0aeb 100644 --- a/src/node/hooks/express/webaccess.js +++ b/src/node/hooks/express/webaccess.js @@ -106,18 +106,22 @@ const checkAccess = async (req, res, next) => { // /////////////////////////////////////////////////////////////////////////////////////////////// let results; + let skip = false; + const preAuthorizeNext = (...args) => { skip = true; next(...args); }; try { - results = await aCallFirst('preAuthorize', {req, res, next}, + results = await aCallFirst('preAuthorize', {req, res, next: preAuthorizeNext}, // This predicate will cause aCallFirst to call the hook functions one at a time until one // of them returns a non-empty list, with an exception: If the request is for an /admin // page, truthy entries are filtered out before checking to see whether the list is empty. // This prevents plugin authors from accidentally granting admin privileges to the general // public. - (r) => (r != null && r.filter((x) => (!requireAdmin || !x)).length > 0)); + (r) => (skip || (r != null && r.filter((x) => (!requireAdmin || !x)).length > 0))); } catch (err) { httpLogger.error(`Error in preAuthorize hook: ${err.stack || err.toString()}`); - return res.status(500).send('Internal Server Error'); + if (!skip) res.status(500).send('Internal Server Error'); + return; } + if (skip) return; if (staticPathsRE.test(req.path)) results.push(true); if (requireAdmin) { // Filter out all 'true' entries to prevent plugin authors from accidentally granting admin diff --git a/src/tests/backend/specs/webaccess.js b/src/tests/backend/specs/webaccess.js index 7594b57e3..70a220aef 100644 --- a/src/tests/backend/specs/webaccess.js +++ b/src/tests/backend/specs/webaccess.js @@ -135,7 +135,7 @@ describe(__filename, function () { assert(!this.called); this.called = true; callOrder.push(this.id); - return cb(this.innerHandle(context.req)); + return cb(this.innerHandle(context)); } }; const handlers = {}; @@ -179,6 +179,13 @@ describe(__filename, function () { await agent.get('/').expect(403); assert.deepEqual(callOrder, ['preAuthorize_0']); }); + it('bypasses authenticate and authorize hooks when next is called', async function () { + settings.requireAuthentication = true; + settings.requireAuthorization = true; + handlers.preAuthorize[0].innerHandle = ({next}) => next(); + await agent.get('/').expect(200); + assert.deepEqual(callOrder, ['preAuthorize_0']); + }); it('bypasses authenticate and authorize hooks for static content, defers', async function () { settings.requireAuthentication = true; settings.requireAuthorization = true; @@ -251,13 +258,13 @@ describe(__filename, function () { 'authenticate_1']); }); it('does not defer if return [true], 200', async function () { - handlers.authenticate[0].innerHandle = (req) => { req.session.user = {}; return [true]; }; + handlers.authenticate[0].innerHandle = ({req}) => { req.session.user = {}; return [true]; }; await agent.get('/').expect(200); // Note: authenticate_1 was not called because authenticate_0 handled it. assert.deepEqual(callOrder, ['preAuthorize_0', 'preAuthorize_1', 'authenticate_0']); }); it('does not defer if return [false], 401', async function () { - handlers.authenticate[0].innerHandle = (req) => [false]; + handlers.authenticate[0].innerHandle = () => [false]; await agent.get('/').expect(401); // Note: authenticate_1 was not called because authenticate_0 handled it. assert.deepEqual(callOrder, ['preAuthorize_0', 'preAuthorize_1', 'authenticate_0']); @@ -355,7 +362,7 @@ describe(__filename, function () { 'authorize_0']); }); it('does not defer if return [false], 403', async function () { - handlers.authorize[0].innerHandle = (req) => [false]; + handlers.authorize[0].innerHandle = () => [false]; await agent.get('/').auth('user', 'user-password').expect(403); // Note: authorize_1 was not called because authorize_0 handled it. assert.deepEqual(callOrder, ['preAuthorize_0', From 7f3d0e71f7a8de79f1f9d5ddab78ac49cda055fe Mon Sep 17 00:00:00 2001 From: Richard Hansen Date: Sat, 18 Dec 2021 01:05:31 -0500 Subject: [PATCH 103/446] express: Check access before `expressConfigure` middleware There are no guarantees about the order of execution of hook functions, which means that a plugin's `expressConfigure` hook function could theoretically register a handler/middleware before the access check middleware is registered. If that happens, the plugin's handler would run before the access check, which would be bad. Avoid the problem by explicitly installing the `webaccess.checkAccess` middleware before running the `expressConfigure` hook. --- src/ep.json | 6 ------ src/node/hooks/express.js | 2 ++ src/node/hooks/express/webaccess.js | 9 ++++++--- 3 files changed, 8 insertions(+), 9 deletions(-) diff --git a/src/ep.json b/src/ep.json index b917aa1f3..63942ac17 100644 --- a/src/ep.json +++ b/src/ep.json @@ -50,12 +50,6 @@ "expressCreateServer": "ep_etherpad-lite/node/hooks/express/padurlsanitize" } }, - { - "name": "webaccess", - "hooks": { - "expressConfigure": "ep_etherpad-lite/node/hooks/express/webaccess" - } - }, { "name": "apicalls", "hooks": { diff --git a/src/node/hooks/express.js b/src/node/hooks/express.js index 351ab5bf2..94d914009 100644 --- a/src/node/hooks/express.js +++ b/src/node/hooks/express.js @@ -12,6 +12,7 @@ const SessionStore = require('../db/SessionStore'); const settings = require('../utils/Settings'); const stats = require('../stats'); const util = require('util'); +const webaccess = require('./express/webaccess'); const logger = log4js.getLogger('http'); let serverName; @@ -203,6 +204,7 @@ exports.restartServer = async () => { app.use(exports.sessionMiddleware); app.use(cookieParser(settings.sessionKey, {})); + app.use(webaccess.checkAccess); await Promise.all([ hooks.aCallAll('expressConfigure', {app}), diff --git a/src/node/hooks/express/webaccess.js b/src/node/hooks/express/webaccess.js index 3d47b0aeb..9ab338498 100644 --- a/src/node/hooks/express/webaccess.js +++ b/src/node/hooks/express/webaccess.js @@ -203,7 +203,10 @@ const checkAccess = async (req, res, next) => { res.status(403).send('Forbidden'); }; -exports.expressConfigure = (hookName, args, cb) => { - args.app.use((req, res, next) => { checkAccess(req, res, next).catch(next); }); - return cb(); +/** + * Express middleware to authenticate the user and check authorization. Must be installed after the + * express-session middleware. + */ +exports.checkAccess = (req, res, next) => { + checkAccess(req, res, next).catch((err) => next(err || new Error(err))); }; From bf35dcfc5096a77b19bc2b50261b34c98fce7011 Mon Sep 17 00:00:00 2001 From: Richard Hansen Date: Sat, 18 Dec 2021 16:30:17 -0500 Subject: [PATCH 104/446] webaccess: Move `preAuthorize` to its own middleware --- src/node/hooks/express.js | 2 + src/node/hooks/express/webaccess.js | 116 +++++++++++++++++----------- 2 files changed, 71 insertions(+), 47 deletions(-) diff --git a/src/node/hooks/express.js b/src/node/hooks/express.js index 94d914009..807127a01 100644 --- a/src/node/hooks/express.js +++ b/src/node/hooks/express.js @@ -204,6 +204,8 @@ exports.restartServer = async () => { app.use(exports.sessionMiddleware); app.use(cookieParser(settings.sessionKey, {})); + // If webaccess.preAuthorize explicitly grants access, webaccess.checkAccess will skip all checks. + app.use(webaccess.preAuthorize); app.use(webaccess.checkAccess); await Promise.all([ diff --git a/src/node/hooks/express/webaccess.js b/src/node/hooks/express/webaccess.js index 9ab338498..16d3bb49b 100644 --- a/src/node/hooks/express/webaccess.js +++ b/src/node/hooks/express/webaccess.js @@ -26,6 +26,14 @@ const staticPathsRE = new RegExp(`^/(?:${[ 'tests/frontend(?:/.*)?', ].join('|')})$`); +// Promisified wrapper around hooks.aCallFirst. +const aCallFirst = (hookName, context, pred = null) => new Promise((resolve, reject) => { + hooks.aCallFirst(hookName, context, (err, r) => err != null ? reject(err) : resolve(r), pred); +}); + +const aCallFirst0 = + async (hookName, context, pred = null) => (await aCallFirst(hookName, context, pred))[0]; + exports.normalizeAuthzLevel = (level) => { if (!level) return false; switch (level) { @@ -56,16 +64,56 @@ exports.userCanModify = (padId, req) => { // Exported so that tests can set this to 0 to avoid unnecessary test slowness. exports.authnFailureDelayMs = 1000; +const preAuthorize = async (req, res, next) => { + const requireAdmin = req.path.toLowerCase().startsWith('/admin'); + const locals = res.locals._webaccess = {requireAdmin, skip: false}; + + // /////////////////////////////////////////////////////////////////////////////////////////////// + // Step 1: Check the preAuthorize hook for early permit/deny (permit is only allowed for non-admin + // pages). If any plugin explicitly grants or denies access, skip the remaining steps. Plugins can + // use the preAuthzFailure hook to override the default 403 error. + // /////////////////////////////////////////////////////////////////////////////////////////////// + + let results; + const preAuthorizeNext = (...args) => { locals.skip = true; next(...args); }; + try { + results = await aCallFirst('preAuthorize', {req, res, next: preAuthorizeNext}, + // This predicate will cause aCallFirst to call the hook functions one at a time until one + // of them returns a non-empty list, with an exception: If the request is for an /admin + // page, truthy entries are filtered out before checking to see whether the list is empty. + // This prevents plugin authors from accidentally granting admin privileges to the general + // public. + (r) => (locals.skip || (r != null && r.filter((x) => (!requireAdmin || !x)).length > 0))); + } catch (err) { + httpLogger.error(`Error in preAuthorize hook: ${err.stack || err.toString()}`); + if (!locals.skip) res.status(500).send('Internal Server Error'); + return; + } + if (locals.skip) return; + if (staticPathsRE.test(req.path)) results.push(true); + if (requireAdmin) { + // Filter out all 'true' entries to prevent plugin authors from accidentally granting admin + // privileges to the general public. + results = results.filter((x) => !x); + } + if (results.length > 0) { + // Access was explicitly granted or denied. If any value is false then access is denied. + if (!results.every((x) => x)) { + // Access explicitly denied. + if (await aCallFirst0('preAuthzFailure', {req, res})) return; + // No plugin handled the pre-authentication authorization failure. + return res.status(403).send('Forbidden'); + } + // Access explicitly granted. + locals.skip = true; + return next('route'); + } + next(); +}; + const checkAccess = async (req, res, next) => { - // Promisified wrapper around hooks.aCallFirst. - const aCallFirst = (hookName, context, pred = null) => new Promise((resolve, reject) => { - hooks.aCallFirst(hookName, context, (err, r) => err != null ? reject(err) : resolve(r), pred); - }); - - const aCallFirst0 = - async (hookName, context, pred = null) => (await aCallFirst(hookName, context, pred))[0]; - - const requireAdmin = req.path.toLowerCase().indexOf('/admin') === 0; + const {locals: {_webaccess: {requireAdmin, skip}}} = res; + if (skip) return next('route'); // This helper is used in steps 2 and 4 below, so it may be called twice per access: once before // authentication is checked and once after (if settings.requireAuthorization is true). @@ -99,43 +147,6 @@ const checkAccess = async (req, res, next) => { return await grant(await aCallFirst0('authorize', {req, res, next, resource: req.path})); }; - // /////////////////////////////////////////////////////////////////////////////////////////////// - // Step 1: Check the preAuthorize hook for early permit/deny (permit is only allowed for non-admin - // pages). If any plugin explicitly grants or denies access, skip the remaining steps. Plugins can - // use the preAuthzFailure hook to override the default 403 error. - // /////////////////////////////////////////////////////////////////////////////////////////////// - - let results; - let skip = false; - const preAuthorizeNext = (...args) => { skip = true; next(...args); }; - try { - results = await aCallFirst('preAuthorize', {req, res, next: preAuthorizeNext}, - // This predicate will cause aCallFirst to call the hook functions one at a time until one - // of them returns a non-empty list, with an exception: If the request is for an /admin - // page, truthy entries are filtered out before checking to see whether the list is empty. - // This prevents plugin authors from accidentally granting admin privileges to the general - // public. - (r) => (skip || (r != null && r.filter((x) => (!requireAdmin || !x)).length > 0))); - } catch (err) { - httpLogger.error(`Error in preAuthorize hook: ${err.stack || err.toString()}`); - if (!skip) res.status(500).send('Internal Server Error'); - return; - } - if (skip) return; - if (staticPathsRE.test(req.path)) results.push(true); - if (requireAdmin) { - // Filter out all 'true' entries to prevent plugin authors from accidentally granting admin - // privileges to the general public. - results = results.filter((x) => !x); - } - if (results.length > 0) { - // Access was explicitly granted or denied. If any value is false then access is denied. - if (results.every((x) => x)) return next(); - if (await aCallFirst0('preAuthzFailure', {req, res})) return; - // No plugin handled the pre-authentication authorization failure. - return res.status(403).send('Forbidden'); - } - // /////////////////////////////////////////////////////////////////////////////////////////////// // Step 2: Try to just access the thing. If access fails (perhaps authentication has not yet // completed, or maybe different credentials are required), go to the next step. @@ -203,9 +214,20 @@ const checkAccess = async (req, res, next) => { res.status(403).send('Forbidden'); }; +/** + * Express middleware that allows plugins to explicitly grant/deny access via the `preAuthorize` + * hook before `checkAccess` is run. If access is explicitly granted: + * - `next('route')` will be called, which can be used to bypass later checks + * - `checkAccess` will simply call `next('route')` + */ +exports.preAuthorize = (req, res, next) => { + preAuthorize(req, res, next).catch((err) => next(err || new Error(err))); +}; + /** * Express middleware to authenticate the user and check authorization. Must be installed after the - * express-session middleware. + * express-session middleware. If the request is pre-authorized, this middleware simply calls + * `next('route')`. */ exports.checkAccess = (req, res, next) => { checkAccess(req, res, next).catch((err) => next(err || new Error(err))); From 0b1ec20c5c061c340db2e7feac2229d79d960296 Mon Sep 17 00:00:00 2001 From: Richard Hansen Date: Sat, 18 Dec 2021 16:54:23 -0500 Subject: [PATCH 105/446] express: Move `preAuthorize` middleware before express-session --- src/node/hooks/express.js | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/node/hooks/express.js b/src/node/hooks/express.js index 807127a01..1e2fc4481 100644 --- a/src/node/hooks/express.js +++ b/src/node/hooks/express.js @@ -201,11 +201,11 @@ exports.restartServer = async () => { secure: 'auto', }, }); - app.use(exports.sessionMiddleware); - app.use(cookieParser(settings.sessionKey, {})); // If webaccess.preAuthorize explicitly grants access, webaccess.checkAccess will skip all checks. app.use(webaccess.preAuthorize); + app.use(exports.sessionMiddleware); + app.use(cookieParser(settings.sessionKey, {})); app.use(webaccess.checkAccess); await Promise.all([ From 72f4ae444d34512eb519589d25c3e1c34503e2a6 Mon Sep 17 00:00:00 2001 From: Richard Hansen Date: Fri, 17 Dec 2021 16:29:45 -0500 Subject: [PATCH 106/446] express: New `expressPreSession` server-side hook --- CHANGELOG.md | 1 + doc/api/hooks_server-side.md | 29 +++++++++++++++++++++++++++++ src/node/hooks/express.js | 4 ++++ 3 files changed, 34 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 488d0e947..39ff70f3c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -19,6 +19,7 @@ #### For plugin authors +* New `expressPreSession` server-side hook. * New APIs for processing attributes: `ep_etherpad-lite/static/js/attributes` (low-level API) and `ep_etherpad-lite/static/js/AttributeMap` (high-level API). diff --git a/doc/api/hooks_server-side.md b/doc/api/hooks_server-side.md index ddae10d14..47477216c 100644 --- a/doc/api/hooks_server-side.md +++ b/doc/api/hooks_server-side.md @@ -58,6 +58,35 @@ Run during startup after the named plugin is initialized. Context properties: None +## `expressPreSession` + +Called from: `src/node/hooks/express.js` + +Called during server startup just before the +[`express-session`](https://www.npmjs.com/package/express-session) middleware is +added to the Express Application object. Use this hook to add route handlers or +middleware that executes before `express-session` state is created and +authentication is performed. This is useful for creating public endpoints that +don't spam the database with new `express-session` records or trigger +authentication. + +**WARNING:** All handlers registered during this hook run before the built-in +authentication checks, so any handled endpoints will be public unless the +handler itself authenticates the user. + +Context properties: + +* `app`: The Express [Application](https://expressjs.com/en/4x/api.html#app) + object. + +Example: + +```javascript +exports.expressPreSession = async (hookName, {app}) => { + app.get('/hello-world', (req, res) => res.send('hello world')); +}; +``` + ## `expressConfigure` Called from: `src/node/hooks/express.js` diff --git a/src/node/hooks/express.js b/src/node/hooks/express.js index 1e2fc4481..2441b827e 100644 --- a/src/node/hooks/express.js +++ b/src/node/hooks/express.js @@ -204,6 +204,10 @@ exports.restartServer = async () => { // If webaccess.preAuthorize explicitly grants access, webaccess.checkAccess will skip all checks. app.use(webaccess.preAuthorize); + // Give plugins an opportunity to install handlers/middleware after the preAuthorize middleware + // but before the express-session middleware. This allows plugins to avoid creating an + // express-session record in the database when it is not needed (e.g., public static content). + await hooks.aCallAll('expressPreSession', {app}); app.use(exports.sessionMiddleware); app.use(cookieParser(settings.sessionKey, {})); app.use(webaccess.checkAccess); From 649fbdccf52cf3d1fd72f4a375dbdefd2d2b5b6a Mon Sep 17 00:00:00 2001 From: Richard Hansen Date: Fri, 17 Dec 2021 17:01:55 -0500 Subject: [PATCH 107/446] express: Move static handlers to `expressPreSession` This avoids the need to exempt the paths from authentication checks, and it eliminates unnecessary express-session state. --- CHANGELOG.md | 3 ++ src/ep.json | 13 ++--- src/node/hooks/express/apicalls.js | 10 ++-- src/node/hooks/express/openapi.js | 5 +- src/node/hooks/express/specialpages.js | 66 +++++++++++++------------- src/node/hooks/express/static.js | 10 ++-- src/node/hooks/express/tests.js | 12 ++--- src/node/hooks/express/webaccess.js | 18 ------- src/node/hooks/i18n.js | 8 ++-- src/tests/backend/specs/webaccess.js | 5 ++ 10 files changed, 65 insertions(+), 85 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 39ff70f3c..4c0b9465f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,9 @@ * `padOptions.showChat` * `padOptions.userColor` * `padOptions.userName` +* Requests for static content (e.g., `/robots.txt`) and special pages (e.g., the + HTTP API, `/stats`) no longer cause the server to generate database records + intended to manage browser sessions (`sessionstorage:*`). * Fixed the return value of the `getText` HTTP API when called with a specific revision. * Fixed a potential attribute pool corruption bug with `copyPadWithoutHistory`. diff --git a/src/ep.json b/src/ep.json index 63942ac17..ec09696c5 100644 --- a/src/ep.json +++ b/src/ep.json @@ -23,7 +23,7 @@ { "name": "static", "hooks": { - "expressCreateServer": "ep_etherpad-lite/node/hooks/express/static" + "expressPreSession": "ep_etherpad-lite/node/hooks/express/static" } }, { @@ -35,13 +35,14 @@ { "name": "i18n", "hooks": { - "expressCreateServer": "ep_etherpad-lite/node/hooks/i18n" + "expressPreSession": "ep_etherpad-lite/node/hooks/i18n" } }, { "name": "specialpages", "hooks": { - "expressCreateServer": "ep_etherpad-lite/node/hooks/express/specialpages" + "expressCreateServer": "ep_etherpad-lite/node/hooks/express/specialpages", + "expressPreSession": "ep_etherpad-lite/node/hooks/express/specialpages" } }, { @@ -53,7 +54,7 @@ { "name": "apicalls", "hooks": { - "expressCreateServer": "ep_etherpad-lite/node/hooks/express/apicalls" + "expressPreSession": "ep_etherpad-lite/node/hooks/express/apicalls" } }, { @@ -79,7 +80,7 @@ { "name": "tests", "hooks": { - "expressCreateServer": "ep_etherpad-lite/node/hooks/express/tests" + "expressPreSession": "ep_etherpad-lite/node/hooks/express/tests" } }, { @@ -105,7 +106,7 @@ { "name": "openapi", "hooks": { - "expressCreateServer": "ep_etherpad-lite/node/hooks/express/openapi" + "expressPreSession": "ep_etherpad-lite/node/hooks/express/openapi" } } ] diff --git a/src/node/hooks/express/apicalls.js b/src/node/hooks/express/apicalls.js index a0fbbc638..834cb4a4b 100644 --- a/src/node/hooks/express/apicalls.js +++ b/src/node/hooks/express/apicalls.js @@ -6,9 +6,9 @@ const formidable = require('formidable'); const apiHandler = require('../../handler/APIHandler'); const util = require('util'); -exports.expressCreateServer = (hookName, args, cb) => { +exports.expressPreSession = async (hookName, {app}) => { // The Etherpad client side sends information about how a disconnect happened - args.app.post('/ep/pad/connection-diagnostic-info', (req, res) => { + app.post('/ep/pad/connection-diagnostic-info', (req, res) => { new formidable.IncomingForm().parse(req, (err, fields, files) => { clientLogger.info(`DIAGNOSTIC-INFO: ${fields.diagnosticInfo}`); res.end('OK'); @@ -23,7 +23,7 @@ exports.expressCreateServer = (hookName, args, cb) => { }); // The Etherpad client side sends information about client side javscript errors - args.app.post('/jserror', (req, res, next) => { + app.post('/jserror', (req, res, next) => { (async () => { const data = JSON.parse(await parseJserrorForm(req)); clientLogger.warn(`${data.msg} --`, { @@ -38,9 +38,7 @@ exports.expressCreateServer = (hookName, args, cb) => { }); // Provide a possibility to query the latest available API version - args.app.get('/api', (req, res) => { + app.get('/api', (req, res) => { res.json({currentVersion: apiHandler.latestApiVersion}); }); - - return cb(); }; diff --git a/src/node/hooks/express/openapi.js b/src/node/hooks/express/openapi.js index c4c1ccf5c..c687b538b 100644 --- a/src/node/hooks/express/openapi.js +++ b/src/node/hooks/express/openapi.js @@ -540,9 +540,7 @@ const generateDefinitionForVersion = (version, style = APIPathStyle.FLAT) => { return definition; }; -exports.expressCreateServer = (hookName, args, cb) => { - const {app} = args; - +exports.expressPreSession = async (hookName, {app}) => { // create openapi-backend handlers for each api version under /api/{version}/* for (const version of Object.keys(apiHandler.version)) { // we support two different styles of api: flat + rest @@ -690,7 +688,6 @@ exports.expressCreateServer = (hookName, args, cb) => { }); } } - return cb(); }; // helper to get api root diff --git a/src/node/hooks/express/specialpages.js b/src/node/hooks/express/specialpages.js index 66ee0221e..8aab5ce4f 100644 --- a/src/node/hooks/express/specialpages.js +++ b/src/node/hooks/express/specialpages.js @@ -10,25 +10,16 @@ const settings = require('../../utils/Settings'); const util = require('util'); const webaccess = require('./webaccess'); -exports.expressCreateServer = (hookName, args, cb) => { - // expose current stats - args.app.get('/stats', (req, res) => { +exports.expressPreSession = async (hookName, {app}) => { + app.get('/stats', (req, res) => { res.json(require('../../stats').toJSON()); }); - // serve index.html under / - args.app.get('/', (req, res) => { - res.send(eejs.require('ep_etherpad-lite/templates/index.html', {req})); - }); - - // serve javascript.html - args.app.get('/javascript', (req, res) => { + app.get('/javascript', (req, res) => { res.send(eejs.require('ep_etherpad-lite/templates/javascript.html', {req})); }); - - // serve robots.txt - args.app.get('/robots.txt', (req, res) => { + app.get('/robots.txt', (req, res) => { let filePath = path.join( settings.root, 'src', @@ -46,6 +37,34 @@ exports.expressCreateServer = (hookName, args, cb) => { }); }); + app.get('/favicon.ico', (req, res, next) => { + (async () => { + const fns = [ + ...(settings.favicon ? [path.resolve(settings.root, settings.favicon)] : []), + path.join(settings.root, 'src', 'static', 'skins', settings.skinName, 'favicon.ico'), + path.join(settings.root, 'src', 'static', 'favicon.ico'), + ]; + for (const fn of fns) { + try { + await fsp.access(fn, fs.constants.R_OK); + } catch (err) { + continue; + } + res.setHeader('Cache-Control', `public, max-age=${settings.maxAge}`); + await util.promisify(res.sendFile.bind(res))(fn); + return; + } + next(); + })().catch((err) => next(err || new Error(err))); + }); +}; + +exports.expressCreateServer = (hookName, args, cb) => { + // serve index.html under / + args.app.get('/', (req, res) => { + res.send(eejs.require('ep_etherpad-lite/templates/index.html', {req})); + }); + // serve pad.html under /p args.app.get('/p/:pad', (req, res, next) => { // The below might break for pads being rewritten @@ -77,26 +96,5 @@ exports.expressCreateServer = (hookName, args, cb) => { })); }); - args.app.get('/favicon.ico', (req, res, next) => { - (async () => { - const fns = [ - ...(settings.favicon ? [path.resolve(settings.root, settings.favicon)] : []), - path.join(settings.root, 'src', 'static', 'skins', settings.skinName, 'favicon.ico'), - path.join(settings.root, 'src', 'static', 'favicon.ico'), - ]; - for (const fn of fns) { - try { - await fsp.access(fn, fs.constants.R_OK); - } catch (err) { - continue; - } - res.setHeader('Cache-Control', `public, max-age=${settings.maxAge}`); - await util.promisify(res.sendFile.bind(res))(fn); - return; - } - next(); - })().catch((err) => next(err || new Error(err))); - }); - return cb(); }; diff --git a/src/node/hooks/express/static.js b/src/node/hooks/express/static.js index 2b01f84cf..26c18995a 100644 --- a/src/node/hooks/express/static.js +++ b/src/node/hooks/express/static.js @@ -28,14 +28,14 @@ const getTar = async () => { return tar; }; -exports.expressCreateServer = async (hookName, args) => { +exports.expressPreSession = async (hookName, {app}) => { // Cache both minified and static. const assetCache = new CachingMiddleware(); - args.app.all(/\/javascripts\/(.*)/, assetCache.handle.bind(assetCache)); + app.all(/\/javascripts\/(.*)/, assetCache.handle.bind(assetCache)); // Minify will serve static files compressed (minify enabled). It also has // file-specific hacks for ace/require-kernel/etc. - args.app.all('/static/:filename(*)', minify.minify); + app.all('/static/:filename(*)', minify.minify); // Setup middleware that will package JavaScript files served by minify for // CommonJS loader on the client-side. @@ -53,12 +53,12 @@ exports.expressCreateServer = async (hookName, args) => { const associator = new StaticAssociator(associations); jsServer.setAssociator(associator); - args.app.use(jsServer.handle.bind(jsServer)); + app.use(jsServer.handle.bind(jsServer)); // serve plugin definitions // not very static, but served here so that client can do // require("pluginfw/static/js/plugin-definitions.js"); - args.app.get('/pluginfw/plugin-definitions.json', (req, res, next) => { + app.get('/pluginfw/plugin-definitions.json', (req, res, next) => { const clientParts = plugins.parts.filter((part) => part.client_hooks != null); const clientPlugins = {}; for (const name of new Set(clientParts.map((part) => part.plugin))) { diff --git a/src/node/hooks/express/tests.js b/src/node/hooks/express/tests.js index 1b1fe8f55..66b47d2af 100644 --- a/src/node/hooks/express/tests.js +++ b/src/node/hooks/express/tests.js @@ -29,8 +29,8 @@ const findSpecs = async (specDir) => { return specs; }; -exports.expressCreateServer = (hookName, args, cb) => { - args.app.get('/tests/frontend/frontendTestSpecs.json', (req, res, next) => { +exports.expressPreSession = async (hookName, {app}) => { + app.get('/tests/frontend/frontendTestSpecs.json', (req, res, next) => { (async () => { const modules = []; await Promise.all(Object.entries(plugins.plugins).map(async ([plugin, def]) => { @@ -59,14 +59,14 @@ exports.expressCreateServer = (hookName, args, cb) => { const rootTestFolder = path.join(settings.root, 'src/tests/frontend/'); - args.app.get('/tests/frontend/index.html', (req, res) => { + app.get('/tests/frontend/index.html', (req, res) => { res.redirect(['./', ...req.url.split('?').slice(1)].join('?')); }); // The regexp /[\d\D]{0,}/ is equivalent to the regexp /.*/. The Express route path used here // uses the more verbose /[\d\D]{0,}/ pattern instead of /.*/ because path-to-regexp v0.1.7 (the // version used with Express v4.x) interprets '.' and '*' differently than regexp. - args.app.get('/tests/frontend/:file([\\d\\D]{0,})', (req, res, next) => { + app.get('/tests/frontend/:file([\\d\\D]{0,})', (req, res, next) => { (async () => { let file = sanitizePathname(req.params.file); if (['', '.', './'].includes(file)) file = 'index.html'; @@ -74,9 +74,7 @@ exports.expressCreateServer = (hookName, args, cb) => { })().catch((err) => next(err || new Error(err))); }); - args.app.get('/tests/frontend', (req, res) => { + app.get('/tests/frontend', (req, res) => { res.redirect(['./frontend/', ...req.url.split('?').slice(1)].join('?')); }); - - return cb(); }; diff --git a/src/node/hooks/express/webaccess.js b/src/node/hooks/express/webaccess.js index 16d3bb49b..42586b757 100644 --- a/src/node/hooks/express/webaccess.js +++ b/src/node/hooks/express/webaccess.js @@ -9,23 +9,6 @@ const readOnlyManager = require('../../db/ReadOnlyManager'); hooks.deprecationNotices.authFailure = 'use the authnFailure and authzFailure hooks instead'; -const staticPathsRE = new RegExp(`^/(?:${[ - 'api(?:/.*)?', - 'favicon\\.ico', - 'ep/pad/connection-diagnostic-info', - 'javascript', - 'javascripts/.*', - 'jserror/?', - 'locales\\.json', - 'locales/.*', - 'rest/.*', - 'pluginfw/.*', - 'robots.txt', - 'static/.*', - 'stats/?', - 'tests/frontend(?:/.*)?', -].join('|')})$`); - // Promisified wrapper around hooks.aCallFirst. const aCallFirst = (hookName, context, pred = null) => new Promise((resolve, reject) => { hooks.aCallFirst(hookName, context, (err, r) => err != null ? reject(err) : resolve(r), pred); @@ -90,7 +73,6 @@ const preAuthorize = async (req, res, next) => { return; } if (locals.skip) return; - if (staticPathsRE.test(req.path)) results.push(true); if (requireAdmin) { // Filter out all 'true' entries to prevent plugin authors from accidentally granting admin // privileges to the general public. diff --git a/src/node/hooks/i18n.js b/src/node/hooks/i18n.js index 1cd663c4d..76487fc59 100644 --- a/src/node/hooks/i18n.js +++ b/src/node/hooks/i18n.js @@ -100,13 +100,13 @@ const generateLocaleIndex = (locales) => { }; -exports.expressCreateServer = (n, args, cb) => { +exports.expressPreSession = async (hookName, {app}) => { // regenerate locales on server restart const locales = getAllLocales(); const localeIndex = generateLocaleIndex(locales); exports.availableLangs = getAvailableLangs(locales); - args.app.get('/locales/:locale', (req, res) => { + app.get('/locales/:locale', (req, res) => { // works with /locale/en and /locale/en.json requests const locale = req.params.locale.split('.')[0]; if (Object.prototype.hasOwnProperty.call(exports.availableLangs, locale)) { @@ -118,11 +118,9 @@ exports.expressCreateServer = (n, args, cb) => { } }); - args.app.get('/locales.json', (req, res) => { + app.get('/locales.json', (req, res) => { res.setHeader('Cache-Control', `public, max-age=${settings.maxAge}`); res.setHeader('Content-Type', 'application/json; charset=utf-8'); res.send(localeIndex); }); - - return cb(); }; diff --git a/src/tests/backend/specs/webaccess.js b/src/tests/backend/specs/webaccess.js index 70a220aef..c55c98ab5 100644 --- a/src/tests/backend/specs/webaccess.js +++ b/src/tests/backend/specs/webaccess.js @@ -77,6 +77,11 @@ describe(__filename, function () { settings.requireAuthorization = false; await agent.get('/admin/').auth('admin', 'admin-password').expect(200); }); + it('authn authz anonymous /robots.txt -> 200', async function () { + settings.requireAuthentication = true; + settings.requireAuthorization = true; + await agent.get('/robots.txt').expect(200); + }); it('authn authz user / -> 403', async function () { settings.requireAuthentication = true; settings.requireAuthorization = true; From 30544b564eb7658c6830ba38a31af856fd6509fc Mon Sep 17 00:00:00 2001 From: Richard Hansen Date: Sat, 18 Dec 2021 17:00:02 -0500 Subject: [PATCH 108/446] express: Skip express-session middleware if pre-authorized --- src/node/hooks/express.js | 12 ++++++++---- src/node/hooks/express/webaccess.js | 10 ++++++++++ 2 files changed, 18 insertions(+), 4 deletions(-) diff --git a/src/node/hooks/express.js b/src/node/hooks/express.js index 2441b827e..8704835a9 100644 --- a/src/node/hooks/express.js +++ b/src/node/hooks/express.js @@ -202,15 +202,19 @@ exports.restartServer = async () => { }, }); - // If webaccess.preAuthorize explicitly grants access, webaccess.checkAccess will skip all checks. app.use(webaccess.preAuthorize); // Give plugins an opportunity to install handlers/middleware after the preAuthorize middleware // but before the express-session middleware. This allows plugins to avoid creating an // express-session record in the database when it is not needed (e.g., public static content). await hooks.aCallAll('expressPreSession', {app}); - app.use(exports.sessionMiddleware); - app.use(cookieParser(settings.sessionKey, {})); - app.use(webaccess.checkAccess); + app.use([ + // If webaccess.preAuthorize explicitly granted access, webaccess.nextRouteIfPreAuthorized will + // call `next('route')` which will skip the remaining middlewares in this list. + webaccess.nextRouteIfPreAuthorized, + exports.sessionMiddleware, + cookieParser(settings.sessionKey, {}), + webaccess.checkAccess, + ]); await Promise.all([ hooks.aCallAll('expressConfigure', {app}), diff --git a/src/node/hooks/express/webaccess.js b/src/node/hooks/express/webaccess.js index 42586b757..10e531717 100644 --- a/src/node/hooks/express/webaccess.js +++ b/src/node/hooks/express/webaccess.js @@ -200,12 +200,22 @@ const checkAccess = async (req, res, next) => { * Express middleware that allows plugins to explicitly grant/deny access via the `preAuthorize` * hook before `checkAccess` is run. If access is explicitly granted: * - `next('route')` will be called, which can be used to bypass later checks + * - `nextRouteIfPreAuthorized` will simply call `next('route')` * - `checkAccess` will simply call `next('route')` */ exports.preAuthorize = (req, res, next) => { preAuthorize(req, res, next).catch((err) => next(err || new Error(err))); }; +/** + * Express middleware that simply calls `next('route')` if the request has been explicitly granted + * access by `preAuthorize` (otherwise it calls `next()`). This can be used to bypass later checks. + */ +exports.nextRouteIfPreAuthorized = (req, res, next) => { + if (res.locals._webaccess.skip) return next('route'); + next(); +}; + /** * Express middleware to authenticate the user and check authorization. Must be installed after the * express-session middleware. If the request is pre-authorized, this middleware simply calls From 2e4c546c7f832d24582ab7f5820a35833c169402 Mon Sep 17 00:00:00 2001 From: Dirk Jagdmann Date: Thu, 9 Dec 2021 20:41:22 -0800 Subject: [PATCH 109/446] Pad: Add new `.spliceText()` method Co-authored-by: Richard Hansen --- src/node/db/Pad.js | 47 ++++++++++++++++++++++++++++++++++++---------- 1 file changed, 37 insertions(+), 10 deletions(-) diff --git a/src/node/db/Pad.js b/src/node/db/Pad.js index cba2f2d74..c17e9b2d1 100644 --- a/src/node/db/Pad.js +++ b/src/node/db/Pad.js @@ -251,30 +251,57 @@ Pad.prototype.getKeyRevisionNumber = function (revNum) { return Math.floor(revNum / 100) * 100; }; +/** + * @returns {string} The pad's text. + */ Pad.prototype.text = function () { return this.atext.text; }; -Pad.prototype.setText = async function (newText) { - newText = exports.cleanText(newText); - if (!newText.endsWith('\n')) newText += '\n'; +/** + * Splices text into the pad. If the result of the splice does not end with a newline, one will be + * automatically appended. + * + * @param {number} start - Location in pad text to start removing and inserting characters. Must be + * a non-negative integer less than or equal to `this.text().length`. + * @param {number} ndel - Number of characters to remove starting at `start`. Must be a non-negative + * integer less than or equal to `this.text().length - start`. + * @param {string} ins - New text to insert at `start` (after the `ndel` characters are deleted). + */ +Pad.prototype.spliceText = async function (start, ndel, ins) { + if (start < 0) throw new RangeError(`start index must be non-negative (is ${start})`); + if (ndel < 0) throw new RangeError(`characters to delete must be non-negative (is ${ndel})`); const orig = this.text(); - if (newText === orig) return; - const changeset = Changeset.makeSplice(orig, 0, orig.length, newText); + assert(orig.endsWith('\n')); + if (start + ndel > orig.length) throw new RangeError('start/delete past the end of the text'); + ins = exports.cleanText(ins); + const willEndWithNewline = + start + ndel < orig.length || // Keeping last char (which is guaranteed to be a newline). + ins.endsWith('\n') || + (!ins && start > 0 && orig[start - 1] === '\n'); + if (!willEndWithNewline) ins += '\n'; + if (ndel === 0 && ins.length === 0) return; + const changeset = Changeset.makeSplice(orig, start, ndel, ins); await this.appendRevision(changeset); }; +/** + * Replaces the pad's text with new text. + * + * @param {string} newText - The pad's new text. If this string does not end with a newline, one + * will be automatically appended. + */ +Pad.prototype.setText = async function (newText) { + await this.spliceText(0, this.text().length, newText); +}; + /** * Appends text to the pad. * * @param {string} newText - Text to insert just BEFORE the pad's existing terminating newline. */ Pad.prototype.appendText = async function (newText) { - newText = exports.cleanText(newText); - const orig = this.text(); - assert(orig.endsWith('\n')); - const changeset = Changeset.makeSplice(orig, orig.length - 1, 0, newText); - await this.appendRevision(changeset); + await this.spliceText(this.text().length - 1, 0, newText); }; /** From 696f9c3367a7850222acbc01f08ae4473e7a81aa Mon Sep 17 00:00:00 2001 From: Richard Hansen Date: Mon, 20 Dec 2021 20:34:43 -0500 Subject: [PATCH 110/446] specialpages: New `/health` endpoint for health checking This endpoint is intended to conform with: https://www.ietf.org/archive/id/draft-inadarei-api-health-check-06.html --- CHANGELOG.md | 2 + src/node/hooks/express/specialpages.js | 10 +++++ src/tests/backend/specs/health.js | 56 ++++++++++++++++++++++++++ 3 files changed, 68 insertions(+) create mode 100644 src/tests/backend/specs/health.js diff --git a/CHANGELOG.md b/CHANGELOG.md index 4c0b9465f..217992cae 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -19,6 +19,8 @@ * Fixed race conditions in the `setText`, `appendText`, and `restoreRevision` functions (HTTP API). * Fixed a crash if the database is busy enough to cause a query timeout. +* New `/health` endpoint for getting information about Etherpad's health (see + [draft-inadarei-api-health-check-06](https://www.ietf.org/archive/id/draft-inadarei-api-health-check-06.html)). #### For plugin authors diff --git a/src/node/hooks/express/specialpages.js b/src/node/hooks/express/specialpages.js index 8aab5ce4f..bf23487c2 100644 --- a/src/node/hooks/express/specialpages.js +++ b/src/node/hooks/express/specialpages.js @@ -11,6 +11,16 @@ const util = require('util'); const webaccess = require('./webaccess'); exports.expressPreSession = async (hookName, {app}) => { + // This endpoint is intended to conform to: + // https://www.ietf.org/archive/id/draft-inadarei-api-health-check-06.html + app.get('/health', (req, res) => { + res.set('Content-Type', 'application/health+json'); + res.json({ + status: 'pass', + releaseId: settings.getEpVersion(), + }); + }); + app.get('/stats', (req, res) => { res.json(require('../../stats').toJSON()); }); diff --git a/src/tests/backend/specs/health.js b/src/tests/backend/specs/health.js new file mode 100644 index 000000000..0090aedbb --- /dev/null +++ b/src/tests/backend/specs/health.js @@ -0,0 +1,56 @@ +'use strict'; + +const assert = require('assert').strict; +const common = require('../common'); +const settings = require('../../../node/utils/Settings'); +const superagent = require('superagent'); + +describe(__filename, function () { + let agent; + const backup = {}; + + const getHealth = () => agent.get('/health') + .accept('application/health+json') + .buffer(true) + .parse(superagent.parse['application/json']) + .expect(200) + .expect((res) => assert.equal(res.type, 'application/health+json')); + + before(async function () { + agent = await common.init(); + }); + + beforeEach(async function () { + backup.settings = {}; + for (const setting of ['requireAuthentication', 'requireAuthorization']) { + backup.settings[setting] = settings[setting]; + } + }); + + afterEach(async function () { + Object.assign(settings, backup.settings); + }); + + it('/health works', async function () { + const res = await getHealth(); + assert.equal(res.body.status, 'pass'); + assert.equal(res.body.releaseId, settings.getEpVersion()); + }); + + it('auth is not required', async function () { + settings.requireAuthentication = true; + settings.requireAuthorization = true; + const res = await getHealth(); + assert.equal(res.body.status, 'pass'); + }); + + // We actually want to test that no express-session state is created, but that is difficult to do + // without intrusive changes or unpleasant ueberdb digging. Instead, we assume that the lack of a + // cookie means that no express-session state was created (how would express-session look up the + // session state if no ID was returned to the client?). + it('no cookie is returned', async function () { + const res = await getHealth(); + const cookie = res.headers['set-cookie']; + assert(cookie == null, `unexpected Set-Cookie: ${cookie}`); + }); +}); From 83f2898723ea72d422425ac727653658dab1bab0 Mon Sep 17 00:00:00 2001 From: Richard Hansen Date: Tue, 21 Dec 2021 01:09:18 -0500 Subject: [PATCH 111/446] package.json: Define `etherpad` binary --- src/package.json | 1 + 1 file changed, 1 insertion(+) diff --git a/src/package.json b/src/package.json index db8f5ad3c..dbc5ab357 100644 --- a/src/package.json +++ b/src/package.json @@ -73,6 +73,7 @@ "wtfnode": "^0.9.1" }, "bin": { + "etherpad": "node/server.js", "etherpad-lite": "node/server.js" }, "devDependencies": { From 11de525508e952475228ab0550cb3d94d5dbc89f Mon Sep 17 00:00:00 2001 From: Richard Hansen Date: Tue, 21 Dec 2021 01:08:14 -0500 Subject: [PATCH 112/446] Docker: Install and use link for `etherpad` binary --- Dockerfile | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/Dockerfile b/Dockerfile index 34c16d165..175c46c2c 100644 --- a/Dockerfile +++ b/Dockerfile @@ -96,7 +96,11 @@ COPY --chown=etherpad:etherpad ./settings.json.docker "${EP_DIR}"/settings.json # Fix group permissions RUN chmod -R g=u . +USER root +RUN cd src && npm link +USER etherpad + HEALTHCHECK --interval=20s --timeout=3s CMD curl -f http://localhost:9001 || exit 1 EXPOSE 9001 -CMD ["node", "src/node/server.js"] +CMD ["etherpad"] From f1856cf95a6291d4a2bd51bed62d6e4b4f6c3ef3 Mon Sep 17 00:00:00 2001 From: Richard Hansen Date: Tue, 21 Dec 2021 01:14:44 -0500 Subject: [PATCH 113/446] Docker: Use new `/health` endpoint for HEALTHCHECK --- CHANGELOG.md | 3 +++ Dockerfile | 3 +-- src/bin/etherpad-healthcheck | 26 +++++++++++++++++++++ src/package-lock.json | 45 ++++++++++++++---------------------- src/package.json | 3 ++- 5 files changed, 49 insertions(+), 31 deletions(-) create mode 100755 src/bin/etherpad-healthcheck diff --git a/CHANGELOG.md b/CHANGELOG.md index 217992cae..87bf6d203 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -21,6 +21,9 @@ * Fixed a crash if the database is busy enough to cause a query timeout. * New `/health` endpoint for getting information about Etherpad's health (see [draft-inadarei-api-health-check-06](https://www.ietf.org/archive/id/draft-inadarei-api-health-check-06.html)). +* Docker now uses the new `/health` endpoint for health checks, which avoids + issues when authentication is enabled. It also avoids the unnecessary creation + of database records for managing browser sessions. #### For plugin authors diff --git a/Dockerfile b/Dockerfile index 175c46c2c..49efb7a70 100644 --- a/Dockerfile +++ b/Dockerfile @@ -64,7 +64,6 @@ RUN export DEBIAN_FRONTEND=noninteractive; \ apt-get -qq --no-install-recommends install \ ca-certificates \ git \ - curl \ ${INSTALL_ABIWORD:+abiword} \ ${INSTALL_SOFFICE:+libreoffice} \ && \ @@ -100,7 +99,7 @@ USER root RUN cd src && npm link USER etherpad -HEALTHCHECK --interval=20s --timeout=3s CMD curl -f http://localhost:9001 || exit 1 +HEALTHCHECK --interval=20s --timeout=3s CMD ["etherpad-healthcheck"] EXPOSE 9001 CMD ["etherpad"] diff --git a/src/bin/etherpad-healthcheck b/src/bin/etherpad-healthcheck new file mode 100755 index 000000000..59105d38a --- /dev/null +++ b/src/bin/etherpad-healthcheck @@ -0,0 +1,26 @@ +#!/usr/bin/env node + +// Checks the health of Etherpad by visiting http://localhost:9001/health. Returns 0 on success, 1 +// on error as required by the Dockerfile HEALTHCHECK instruction. + +'use strict'; + +// As of v14, Node.js does not exit when there is an unhandled Promise rejection. Convert an +// unhandled rejection into an uncaught exception, which does cause Node.js to exit. +process.on('unhandledRejection', (err) => { throw err; }); + +const assert = require('assert').strict; +const superagent = require('superagent'); + +(async () => { + const res = await superagent.get('http://localhost:9001/health') + .accept('application/health+json') + .buffer(true) + .parse(superagent.parse['application/json']); + assert(res.ok, `Unexpected HTTP status: ${res.status}`); + assert.equal(res.type, 'application/health+json'); + const {body: {status} = {}} = res; + assert(status != null); + assert.equal(typeof status, 'string'); + assert(['pass', 'ok', 'up'].includes(status.toLowerCase()), `Unexpected status: ${status}`); +})(); diff --git a/src/package-lock.json b/src/package-lock.json index 22304dde4..d1f16f8e7 100644 --- a/src/package-lock.json +++ b/src/package-lock.json @@ -1460,10 +1460,9 @@ "integrity": "sha1-4wOogrNCzD7oylE6eZmXNNqzriw=" }, "cookiejar": { - "version": "2.1.2", - "resolved": "https://registry.npmjs.org/cookiejar/-/cookiejar-2.1.2.tgz", - "integrity": "sha512-Mw+adcfzPxcPeI+0WlvRrr/3lGVO0bD75SxX6811cxSh1Wbxx7xZBGK1eVtDf6si8rg2lhnUjsVLMFMfbRIuwA==", - "dev": true + "version": "2.1.3", + "resolved": "https://registry.npmjs.org/cookiejar/-/cookiejar-2.1.3.tgz", + "integrity": "sha512-JxbCBUdrfr6AQjOXrxoTvAMJO4HBTUIlBzslcJPAz+/KT8yk53fXun51u+RenNYvad/+Vc2DIz5o9UxlCDymFQ==" }, "core-util-is": { "version": "1.0.2", @@ -2437,10 +2436,9 @@ "integrity": "sha1-PYpcZog6FqMMqGQ+hR8Zuqd5eRc=" }, "fast-safe-stringify": { - "version": "2.0.8", - "resolved": "https://registry.npmjs.org/fast-safe-stringify/-/fast-safe-stringify-2.0.8.tgz", - "integrity": "sha512-lXatBjf3WPjmWD6DpIZxkeSsCOwqI0maYMpgDlx8g4U2qi4lbjA9oH/HD2a87G+KfsUmo5WbJFmqBZlPxtptag==", - "dev": true + "version": "2.1.1", + "resolved": "https://registry.npmjs.org/fast-safe-stringify/-/fast-safe-stringify-2.1.1.tgz", + "integrity": "sha512-W+KJc2dmILlPplD/H4K9l9LcAHAfPtP6BY84uVLXQ6Evcz9Lcg33Y2z1IVblT6xdY54PXYVHEv+0Wpq8Io6zkA==" }, "file-entry-cache": { "version": "6.0.1", @@ -8507,7 +8505,6 @@ "version": "6.1.0", "resolved": "https://registry.npmjs.org/superagent/-/superagent-6.1.0.tgz", "integrity": "sha512-OUDHEssirmplo3F+1HWKUrUjvnQuA+nZI6i/JJBdXb5eq9IyEQwPyPpqND+SSsxf6TygpBEkUjISVRN4/VOpeg==", - "dev": true, "requires": { "component-emitter": "^1.3.0", "cookiejar": "^2.1.2", @@ -8523,10 +8520,9 @@ }, "dependencies": { "debug": { - "version": "4.3.2", - "resolved": "https://registry.npmjs.org/debug/-/debug-4.3.2.tgz", - "integrity": "sha512-mOp8wKcvj7XxC78zLgw/ZA+6TSgkoE2C/ienthhRD298T7UNwAg9diBpLRxC0mOezLl4B0xV7M0cCO6P/O0Xhw==", - "dev": true, + "version": "4.3.3", + "resolved": "https://registry.npmjs.org/debug/-/debug-4.3.3.tgz", + "integrity": "sha512-/zxw5+vh1Tfv+4Qn7a5nsbcJKPaSvCDhojn6FEl9vupwK2VCSDtEiEtqr8DFtzYFOdz63LBkxec7DYuc2jon6Q==", "requires": { "ms": "2.1.2" } @@ -8535,7 +8531,6 @@ "version": "3.0.1", "resolved": "https://registry.npmjs.org/form-data/-/form-data-3.0.1.tgz", "integrity": "sha512-RHkBKtLWUVwd7SqRIvCZMEvAMoGUp0XU+seQiZejj0COz3RI3hWP4sCv3gZWWLjJTd7rGwcsF5eKZGii0r/hbg==", - "dev": true, "requires": { "asynckit": "^0.4.0", "combined-stream": "^1.0.8", @@ -8543,22 +8538,19 @@ } }, "mime": { - "version": "2.5.2", - "resolved": "https://registry.npmjs.org/mime/-/mime-2.5.2.tgz", - "integrity": "sha512-tqkh47FzKeCPD2PUiPB6pkbMzsCasjxAfC62/Wap5qrUWcb+sFasXUC5I3gYM5iBM8v/Qpn4UK0x+j0iHyFPDg==", - "dev": true + "version": "2.6.0", + "resolved": "https://registry.npmjs.org/mime/-/mime-2.6.0.tgz", + "integrity": "sha512-USPkMeET31rOMiarsBNIHZKLGgvKc/LrjofAnBlOttf5ajRvqiRA8QsenbcooctK6d6Ts6aqZXBA+XbkKthiQg==" }, "ms": { "version": "2.1.2", "resolved": "https://registry.npmjs.org/ms/-/ms-2.1.2.tgz", - "integrity": "sha512-sGkPx+VjMtmA6MX27oA4FBFELFCZZ4S4XqeGOXCv68tT+jb3vk/RyaKWP0PTKyWtmLSM0b+adUTEvbs1PEaH2w==", - "dev": true + "integrity": "sha512-sGkPx+VjMtmA6MX27oA4FBFELFCZZ4S4XqeGOXCv68tT+jb3vk/RyaKWP0PTKyWtmLSM0b+adUTEvbs1PEaH2w==" }, "qs": { - "version": "6.10.1", - "resolved": "https://registry.npmjs.org/qs/-/qs-6.10.1.tgz", - "integrity": "sha512-M528Hph6wsSVOBiYUnGf+K/7w0hNshs/duGsNXPUCLH5XAqjEtiPGwNONLV0tBH8NoGb0mvD5JubnUTrujKDTg==", - "dev": true, + "version": "6.10.2", + "resolved": "https://registry.npmjs.org/qs/-/qs-6.10.2.tgz", + "integrity": "sha512-mSIdjzqznWgfd4pMii7sHtaYF8rx8861hBO80SraY5GT0XQibWZWJSid0avzHGkDIZLImux2S5mXO0Hfct2QCw==", "requires": { "side-channel": "^1.0.4" } @@ -8567,7 +8559,6 @@ "version": "3.6.0", "resolved": "https://registry.npmjs.org/readable-stream/-/readable-stream-3.6.0.tgz", "integrity": "sha512-BViHy7LKeTz4oNnkcLJ+lVSL6vpiFeX6/d3oSH8zCW7UxP2onchk+vTGB143xuFjHS3deTgkKoXXymXqymiIdA==", - "dev": true, "requires": { "inherits": "^2.0.3", "string_decoder": "^1.1.1", @@ -8577,14 +8568,12 @@ "safe-buffer": { "version": "5.2.1", "resolved": "https://registry.npmjs.org/safe-buffer/-/safe-buffer-5.2.1.tgz", - "integrity": "sha512-rp3So07KcdmmKbGvgaNxQSJr7bGVSVk5S9Eq1F+ppbRo70+YeaDxkw5Dd8NPN+GD6bjnYm2VuPuCXmpuYvmCXQ==", - "dev": true + "integrity": "sha512-rp3So07KcdmmKbGvgaNxQSJr7bGVSVk5S9Eq1F+ppbRo70+YeaDxkw5Dd8NPN+GD6bjnYm2VuPuCXmpuYvmCXQ==" }, "string_decoder": { "version": "1.3.0", "resolved": "https://registry.npmjs.org/string_decoder/-/string_decoder-1.3.0.tgz", "integrity": "sha512-hkRX8U1WjJFd8LsDJ2yQ/wWWxaopEsABU1XfkM8A+j0+85JAGppt16cr1Whg6KIbb4okU6Mql6BOj+uup/wKeA==", - "dev": true, "requires": { "safe-buffer": "~5.2.0" } diff --git a/src/package.json b/src/package.json index dbc5ab357..1565939aa 100644 --- a/src/package.json +++ b/src/package.json @@ -63,6 +63,7 @@ "security": "1.0.0", "semver": "^7.3.5", "socket.io": "^2.4.1", + "superagent": "^6.1.0", "terser": "^5.10.0", "threads": "^1.7.0", "tiny-worker": "^2.3.0", @@ -74,6 +75,7 @@ }, "bin": { "etherpad": "node/server.js", + "etherpad-healthcheck": "bin/etherpad-healthcheck", "etherpad-lite": "node/server.js" }, "devDependencies": { @@ -95,7 +97,6 @@ "set-cookie-parser": "^2.4.8", "sinon": "^12.0.1", "split-grid": "^1.0.11", - "superagent": "^6.1.0", "supertest": "^6.1.6" }, "eslintConfig": { From 8539a664396d965426fb3050947a11f3609850eb Mon Sep 17 00:00:00 2001 From: Richard Hansen Date: Mon, 20 Dec 2021 17:27:55 -0500 Subject: [PATCH 114/446] docs: Improve `handleMessageSecurity` documentation --- doc/api/hooks_server-side.md | 80 ++++++++++++++---------------------- 1 file changed, 31 insertions(+), 49 deletions(-) diff --git a/doc/api/hooks_server-side.md b/doc/api/hooks_server-side.md index 47477216c..648aff224 100644 --- a/doc/api/hooks_server-side.md +++ b/doc/api/hooks_server-side.md @@ -574,26 +574,23 @@ exports.authzFailure = (hookName, context, cb) => { }; ``` -## handleMessage -Called from: src/node/handler/PadMessageHandler.js +## `handleMessage` -Things in context: - -1. message - the message being handled -2. socket - the socket.io Socket object -3. client - **deprecated** synonym of socket +Called from: `src/node/handler/PadMessageHandler.js` This hook allows plugins to drop or modify incoming socket.io messages from -clients, before Etherpad processes them. +clients, before Etherpad processes them. If any hook function returns `null` +then the message will not be subject to further processing. -The handleMessage function must return a Promise. If the Promise resolves to -`null`, the message is dropped. Returning `callback(value)` will return a -Promise that is resolved to `value`. +Context properties: -Examples: +* `message`: The message being handled. +* `socket`: The socket.io Socket object. +* `client`: (**Deprecated**; use `socket` instead.) Synonym of `socket`. -``` -// Using an async function: +Example: + +```javascript exports.handleMessage = async (hookName, {message, socket}) => { if (message.type === 'USERINFO_UPDATE') { // Force the display name to the name associated with the account. @@ -601,51 +598,36 @@ exports.handleMessage = async (hookName, {message, socket}) => { if (user.name) message.data.userInfo.name = user.name; } }; - -// Using a regular function: -exports.handleMessage = (hookName, {message, socket}, callback) => { - if (message.type === 'USERINFO_UPDATE') { - // Force the display name to the name associated with the account. - const user = socket.client.request.session.user || {}; - if (user.name) message.data.userInfo.name = user.name; - } - return callback(); -}; ``` -## handleMessageSecurity -Called from: src/node/handler/PadMessageHandler.js +## `handleMessageSecurity` -Things in context: +Called from: `src/node/handler/PadMessageHandler.js` -1. message - the message being handled -2. socket - the socket.io Socket object -3. client - **deprecated** synonym of socket +Called for each incoming message from a client. Allows plugins to grant +temporary write access to a pad. -This hook allows plugins to grant temporary write access to a pad. It is called -for each incoming message from a client. If write access is granted, it applies -to the current message and all future messages from the same socket.io -connection until the next `CLIENT_READY` message. Read-only access is reset -**after** each `CLIENT_READY` message, so granting write access has no effect -for those message types. +Supported return values: -The handleMessageSecurity function must return a Promise. If the Promise -resolves to `true`, write access is granted as described above. Returning -`callback(value)` will return a Promise that is resolved to `value`. +* `undefined`: No change in access status. +* `true`: Override the user's read-only access for all `COLLABROOM` messages + from the same socket.io connection (including the current message, if + applicable) until the client's next `CLIENT_READY` message. Has no effect if + the user already has write access to the pad. Read-only access is reset + **after** each `CLIENT_READY` message, so returning `true` has no effect for + `CLIENT_READY` messages. -Examples: +Context properties: -``` -// Using an async function: +* `message`: The message being handled. +* `socket`: The socket.io Socket object. +* `client`: (**Deprecated**; use `socket` instead.) Synonym of `socket`. + +Example: + +```javascript exports.handleMessageSecurity = async (hookName, {message, socket}) => { if (shouldGrantWriteAccess(message, socket)) return true; - return; -}; - -// Using a regular function: -exports.handleMessageSecurity = (hookName, {message, socket}, callback) => { - if (shouldGrantWriteAccess(message, socket)) return callback(true); - return callback(); }; ``` From 1b52c9f0c4f2f6bd63530899008e67fe440ec2fb Mon Sep 17 00:00:00 2001 From: Richard Hansen Date: Mon, 20 Dec 2021 16:58:39 -0500 Subject: [PATCH 115/446] PadMessageHandler: Deprecate `client` context property --- CHANGELOG.md | 3 +++ src/node/handler/PadMessageHandler.js | 12 +++++++++++- 2 files changed, 14 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 87bf6d203..563d8c40c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -37,6 +37,9 @@ #### For plugin authors +* The `client` context property for the `handleMessageSecurity` and + `handleMessage` server-side hooks is deprecated; use the `socket` context + property instead. * Changes to the `src/static/js/Changeset.js` library: * The following attribute processing functions are deprecated (use the new attribute APIs instead): diff --git a/src/node/handler/PadMessageHandler.js b/src/node/handler/PadMessageHandler.js index 3460983bd..da25b38dd 100644 --- a/src/node/handler/PadMessageHandler.js +++ b/src/node/handler/PadMessageHandler.js @@ -26,6 +26,7 @@ const ChatMessage = require('../../static/js/ChatMessage'); const AttributePool = require('../../static/js/AttributePool'); const AttributeManager = require('../../static/js/AttributeManager'); const authorManager = require('../db/AuthorManager'); +const {padutils} = require('../../static/js/pad_utils'); const readOnlyManager = require('../db/ReadOnlyManager'); const settings = require('../utils/Settings'); const securityManager = require('../db/SecurityManager'); @@ -270,7 +271,16 @@ exports.handleMessage = async (socket, message) => { thisSession.author = authorID; // Allow plugins to bypass the readonly message blocker - const context = {message, socket, client: socket}; // `client` for backwards compatibility. + const context = { + message, + socket, + get client() { + padutils.warnDeprecated( + 'the `client` context property for the handleMessageSecurity and handleMessage hooks ' + + 'is deprecated; use the `socket` property instead'); + return this.socket; + }, + }; if ((await hooks.aCallAll('handleMessageSecurity', context)).some((w) => w === true)) { thisSession.readonly = false; } From 31b025bd9dcaa59b0d820fd254172ddfb214c4f0 Mon Sep 17 00:00:00 2001 From: Richard Hansen Date: Tue, 7 Dec 2021 02:30:08 -0500 Subject: [PATCH 116/446] PadMessageHandler: Pass session info to `handleMessageSecurity` hook --- CHANGELOG.md | 3 +++ doc/api/hooks_server-side.md | 16 +++++++++++++++- src/node/handler/PadMessageHandler.js | 16 ++++++++++------ src/node/hooks/express/webaccess.js | 6 ++---- 4 files changed, 30 insertions(+), 11 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 563d8c40c..0179005a7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -32,6 +32,9 @@ (low-level API) and `ep_etherpad-lite/static/js/AttributeMap` (high-level API). * The `import` server-side hook has a new `ImportError` context property. +* The `handleMessageSecurity` and `handleMessage` server-side hooks have a new + `sessionInfo` context property that includes the user's author ID, the pad ID, + and whether the user only has read-only access. ### Compatibility changes diff --git a/doc/api/hooks_server-side.md b/doc/api/hooks_server-side.md index 648aff224..54192ef3c 100644 --- a/doc/api/hooks_server-side.md +++ b/doc/api/hooks_server-side.md @@ -585,6 +585,12 @@ then the message will not be subject to further processing. Context properties: * `message`: The message being handled. +* `sessionInfo`: Object describing the socket.io session with the following + properties: + * `authorId`: The user's author ID. + * `padId`: The real (not read-only) ID of the pad. + * `readOnly`: Whether the client has read-only access (true) or read/write + access (false). * `socket`: The socket.io Socket object. * `client`: (**Deprecated**; use `socket` instead.) Synonym of `socket`. @@ -620,13 +626,21 @@ Supported return values: Context properties: * `message`: The message being handled. +* `sessionInfo`: Object describing the socket.io connection with the following + properties: + * `authorId`: The user's author ID. + * `padId`: The real (not read-only) ID of the pad. + * `readOnly`: Whether the client has read-only access (true) or read/write + access (false). * `socket`: The socket.io Socket object. * `client`: (**Deprecated**; use `socket` instead.) Synonym of `socket`. Example: ```javascript -exports.handleMessageSecurity = async (hookName, {message, socket}) => { +exports.handleMessageSecurity = async (hookName, context) => { + const {message, sessionInfo: {readOnly}, socket} = context; + if (!readOnly || message.type !== 'COLLABROOM') return; if (shouldGrantWriteAccess(message, socket)) return true; }; ``` diff --git a/src/node/handler/PadMessageHandler.js b/src/node/handler/PadMessageHandler.js index da25b38dd..34db0cbc6 100644 --- a/src/node/handler/PadMessageHandler.js +++ b/src/node/handler/PadMessageHandler.js @@ -235,6 +235,11 @@ exports.handleMessage = async (socket, message) => { padID: message.padId, token: message.token, }; + const padIds = await readOnlyManager.getIds(thisSession.auth.padID); + thisSession.padId = padIds.padId; + thisSession.readOnlyPadId = padIds.readOnlyPadId; + thisSession.readonly = + padIds.readonly || !webaccess.userCanModify(thisSession.auth.padID, socket.client.request); } const auth = thisSession.auth; @@ -273,6 +278,11 @@ exports.handleMessage = async (socket, message) => { // Allow plugins to bypass the readonly message blocker const context = { message, + sessionInfo: { + authorId: thisSession.author, + padId: thisSession.padId, + readOnly: thisSession.readonly, + }, socket, get client() { padutils.warnDeprecated( @@ -793,12 +803,6 @@ const handleClientReady = async (socket, message) => { if (sessionInfo == null) return; assert(sessionInfo.author); - const padIds = await readOnlyManager.getIds(sessionInfo.auth.padID); - sessionInfo.padId = padIds.padId; - sessionInfo.readOnlyPadId = padIds.readOnlyPadId; - sessionInfo.readonly = - padIds.readonly || !webaccess.userCanModify(sessionInfo.auth.padID, socket.client.request); - await hooks.aCallAll('clientReady', message); // Deprecated due to awkward context. let {colorId: authorColorId, name: authorName} = message.userInfo || {}; diff --git a/src/node/hooks/express/webaccess.js b/src/node/hooks/express/webaccess.js index 10e531717..1e1adf41a 100644 --- a/src/node/hooks/express/webaccess.js +++ b/src/node/hooks/express/webaccess.js @@ -36,12 +36,10 @@ exports.userCanModify = (padId, req) => { if (readOnlyManager.isReadOnlyId(padId)) return false; if (!settings.requireAuthentication) return true; const {session: {user} = {}} = req; - assert(user); // If authn required and user == null, the request should have already been denied. - if (user.readOnly) return false; + if (!user || user.readOnly) return false; assert(user.padAuthorizations); // This is populated even if !settings.requireAuthorization. const level = exports.normalizeAuthzLevel(user.padAuthorizations[padId]); - assert(level); // If !level, the request should have already been denied. - return level !== 'readOnly'; + return level && level !== 'readOnly'; }; // Exported so that tests can set this to 0 to avoid unnecessary test slowness. From 02a56dc58c4f4530dfdf819a81637cde9c0f76e6 Mon Sep 17 00:00:00 2001 From: Richard Hansen Date: Mon, 20 Dec 2021 17:55:00 -0500 Subject: [PATCH 117/446] PadMessageHandler: Allow `handleMessageSecurity` to grant one-time write access --- CHANGELOG.md | 4 ++ doc/api/hooks_server-side.md | 19 ++++--- src/node/handler/PadMessageHandler.js | 20 ++++++- src/tests/backend/common.js | 4 +- src/tests/backend/specs/messages.js | 82 ++++++++++++++++++++------- 5 files changed, 94 insertions(+), 35 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 0179005a7..ad489422d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -35,6 +35,8 @@ * The `handleMessageSecurity` and `handleMessage` server-side hooks have a new `sessionInfo` context property that includes the user's author ID, the pad ID, and whether the user only has read-only access. +* The `handleMessageSecurity` server-side hook can now be used to grant write + access for the current message only. ### Compatibility changes @@ -43,6 +45,8 @@ * The `client` context property for the `handleMessageSecurity` and `handleMessage` server-side hooks is deprecated; use the `socket` context property instead. +* Returning `true` from a `handleMessageSecurity` hook function is deprecated; + return `'permitOnce'` instead. * Changes to the `src/static/js/Changeset.js` library: * The following attribute processing functions are deprecated (use the new attribute APIs instead): diff --git a/doc/api/hooks_server-side.md b/doc/api/hooks_server-side.md index 54192ef3c..81e505108 100644 --- a/doc/api/hooks_server-side.md +++ b/doc/api/hooks_server-side.md @@ -616,12 +616,15 @@ temporary write access to a pad. Supported return values: * `undefined`: No change in access status. -* `true`: Override the user's read-only access for all `COLLABROOM` messages - from the same socket.io connection (including the current message, if - applicable) until the client's next `CLIENT_READY` message. Has no effect if - the user already has write access to the pad. Read-only access is reset - **after** each `CLIENT_READY` message, so returning `true` has no effect for - `CLIENT_READY` messages. +* `'permitOnce'`: Override the user's read-only access for the current + `COLLABROOM` message only. Has no effect if the current message is not a + `COLLABROOM` message, or if the user already has write access to the pad. +* `true`: (**Deprecated**; return `'permitOnce'` instead.) Override the user's + read-only access for all `COLLABROOM` messages from the same socket.io + connection (including the current message, if applicable) until the client's + next `CLIENT_READY` message. Has no effect if the user already has write + access to the pad. Read-only access is reset **after** each `CLIENT_READY` + message, so returning `true` has no effect for `CLIENT_READY` messages. Context properties: @@ -639,9 +642,9 @@ Example: ```javascript exports.handleMessageSecurity = async (hookName, context) => { - const {message, sessionInfo: {readOnly}, socket} = context; + const {message, sessionInfo: {readOnly}} = context; if (!readOnly || message.type !== 'COLLABROOM') return; - if (shouldGrantWriteAccess(message, socket)) return true; + if (await messageIsBenign(message)) return 'permitOnce'; }; ``` diff --git a/src/node/handler/PadMessageHandler.js b/src/node/handler/PadMessageHandler.js index 34db0cbc6..a358807e9 100644 --- a/src/node/handler/PadMessageHandler.js +++ b/src/node/handler/PadMessageHandler.js @@ -276,6 +276,7 @@ exports.handleMessage = async (socket, message) => { thisSession.author = authorID; // Allow plugins to bypass the readonly message blocker + let readOnly = thisSession.readonly; const context = { message, sessionInfo: { @@ -291,8 +292,21 @@ exports.handleMessage = async (socket, message) => { return this.socket; }, }; - if ((await hooks.aCallAll('handleMessageSecurity', context)).some((w) => w === true)) { - thisSession.readonly = false; + for (const res of await hooks.aCallAll('handleMessageSecurity', context)) { + switch (res) { + case true: + padutils.warnDeprecated( + 'returning `true` from a `handleMessageSecurity` hook function is deprecated; ' + + 'return "permitOnce" instead'); + thisSession.readonly = false; + // Fall through: + case 'permitOnce': + readOnly = false; + break; + default: + messageLogger.warn( + 'Ignoring unsupported return value from handleMessageSecurity hook function:', res); + } } // Call handleMessage hook. If a plugin returns null, the message will be dropped. @@ -312,7 +326,7 @@ exports.handleMessage = async (socket, message) => { } else if (message.type === 'CHANGESET_REQ') { await handleChangesetRequest(socket, message); } else if (message.type === 'COLLABROOM') { - if (thisSession.readonly) { + if (readOnly) { messageLogger.warn('Dropped message, COLLABROOM for readonly pad'); } else if (message.data.type === 'USER_CHANGES') { stats.counter('pendingEdits').inc(); diff --git a/src/tests/backend/common.js b/src/tests/backend/common.js index 1787ace7d..000354232 100644 --- a/src/tests/backend/common.js +++ b/src/tests/backend/common.js @@ -172,14 +172,14 @@ exports.connect = async (res = null) => { * @param {string} padId - Which pad to join. * @returns The CLIENT_VARS message from the server. */ -exports.handshake = async (socket, padId) => { +exports.handshake = async (socket, padId, token = 't.12345') => { logger.debug('sending CLIENT_READY...'); socket.send({ component: 'pad', type: 'CLIENT_READY', padId, sessionID: null, - token: 't.12345', + token, }); logger.debug('waiting for CLIENT_VARS response...'); const msg = await exports.waitForSocketEvent(socket, 'message'); diff --git a/src/tests/backend/specs/messages.js b/src/tests/backend/specs/messages.js index 2d5546d6c..e36e8d98b 100644 --- a/src/tests/backend/specs/messages.js +++ b/src/tests/backend/specs/messages.js @@ -1,103 +1,141 @@ 'use strict'; -const AttributePool = require('../../../static/js/AttributePool'); const assert = require('assert').strict; const common = require('../common'); const padManager = require('../../../node/db/PadManager'); +const plugins = require('../../../static/js/pluginfw/plugin_defs'); +const readOnlyManager = require('../../../node/db/ReadOnlyManager'); describe(__filename, function () { let agent; let pad; let padId; + let roPadId; let rev; let socket; + let roSocket; + const backups = {}; before(async function () { agent = await common.init(); }); beforeEach(async function () { + backups.hooks = {handleMessageSecurity: plugins.hooks.handleMessageSecurity}; + plugins.hooks.handleMessageSecurity = []; padId = common.randomString(); assert(!await padManager.doesPadExist(padId)); - pad = await padManager.getPad(padId, ''); + pad = await padManager.getPad(padId, 'dummy text'); + await pad.setText('\n'); // Make sure the pad is created. assert.equal(pad.text(), '\n'); - const res = await agent.get(`/p/${padId}`).expect(200); + let res = await agent.get(`/p/${padId}`).expect(200); socket = await common.connect(res); const {type, data: clientVars} = await common.handshake(socket, padId); assert.equal(type, 'CLIENT_VARS'); rev = clientVars.collab_client_vars.rev; + + roPadId = await readOnlyManager.getReadOnlyId(padId); + res = await agent.get(`/p/${roPadId}`).expect(200); + roSocket = await common.connect(res); + await common.handshake(roSocket, roPadId, `t.${common.randomString(8)}`); }); afterEach(async function () { + Object.assign(plugins.hooks, backups.hooks); if (socket != null) socket.close(); socket = null; + if (roSocket != null) roSocket.close(); + roSocket = null; if (pad != null) await pad.remove(); pad = null; }); describe('USER_CHANGES', function () { const sendUserChanges = - async (changeset) => await common.sendUserChanges(socket, {baseRev: rev, changeset}); - const assertAccepted = async (wantRev) => { + async (socket, cs) => await common.sendUserChanges(socket, {baseRev: rev, changeset: cs}); + const assertAccepted = async (socket, wantRev) => { await common.waitForAcceptCommit(socket, wantRev); rev = wantRev; }; - const assertRejected = async () => { + const assertRejected = async (socket) => { const msg = await common.waitForSocketEvent(socket, 'message'); assert.deepEqual(msg, {disconnect: 'badChangeset'}); }; it('changes are applied', async function () { await Promise.all([ - assertAccepted(rev + 1), - sendUserChanges('Z:1>5+5$hello'), + assertAccepted(socket, rev + 1), + sendUserChanges(socket, 'Z:1>5+5$hello'), ]); assert.equal(pad.text(), 'hello\n'); }); it('bad changeset is rejected', async function () { await Promise.all([ - assertRejected(), - sendUserChanges('this is not a valid changeset'), + assertRejected(socket), + sendUserChanges(socket, 'this is not a valid changeset'), ]); }); it('retransmission is accepted, has no effect', async function () { const cs = 'Z:1>5+5$hello'; await Promise.all([ - assertAccepted(rev + 1), - sendUserChanges(cs), + assertAccepted(socket, rev + 1), + sendUserChanges(socket, cs), ]); --rev; await Promise.all([ - assertAccepted(rev + 1), - sendUserChanges(cs), + assertAccepted(socket, rev + 1), + sendUserChanges(socket, cs), ]); assert.equal(pad.text(), 'hello\n'); }); it('identity changeset is accepted, has no effect', async function () { await Promise.all([ - assertAccepted(rev + 1), - sendUserChanges('Z:1>5+5$hello'), + assertAccepted(socket, rev + 1), + sendUserChanges(socket, 'Z:1>5+5$hello'), ]); await Promise.all([ - assertAccepted(rev), - sendUserChanges('Z:6>0$'), + assertAccepted(socket, rev), + sendUserChanges(socket, 'Z:6>0$'), ]); assert.equal(pad.text(), 'hello\n'); }); it('non-identity changeset with no net change is accepted, has no effect', async function () { await Promise.all([ - assertAccepted(rev + 1), - sendUserChanges('Z:1>5+5$hello'), + assertAccepted(socket, rev + 1), + sendUserChanges(socket, 'Z:1>5+5$hello'), ]); await Promise.all([ - assertAccepted(rev), - sendUserChanges('Z:6>0-5+5$hello'), + assertAccepted(socket, rev), + sendUserChanges(socket, 'Z:6>0-5+5$hello'), ]); assert.equal(pad.text(), 'hello\n'); }); + + it('handleMessageSecurity can grant one-time write access', async function () { + const cs = 'Z:1>5+5$hello'; + // First try to send a change and verify that it was dropped. + await sendUserChanges(roSocket, cs); + // sendUserChanges() waits for message ack, so if the message was accepted then head should + // have already incremented by the time we get here. + assert.equal(pad.head, rev); // Not incremented. + + // Now allow the change. + plugins.hooks.handleMessageSecurity.push({hook_fn: () => 'permitOnce'}); + await Promise.all([ + assertAccepted(roSocket, rev + 1), + sendUserChanges(roSocket, cs), + ]); + assert.equal(pad.text(), 'hello\n'); + + // The next change should be dropped. + plugins.hooks.handleMessageSecurity = []; + await sendUserChanges(roSocket, 'Z:6>6=5+6$ world'); + assert.equal(pad.head, rev); // Not incremented. + assert.equal(pad.text(), 'hello\n'); + }); }); }); From cb257de8f91332ff780ba3acdf0131fd2dde5142 Mon Sep 17 00:00:00 2001 From: Richard Hansen Date: Mon, 20 Dec 2021 19:57:23 -0500 Subject: [PATCH 118/446] Bump version to v1.9.0 for plugin `peerDependencies` This allows plugins to depend on the not-yet-released API by bumping their `peerDependencies` to `>=1.9.0`. IMPORTANT: v1.9.0 IS NOT RELEASED YET. I tried to bump the version to 1.9.0-alpha.0 instead, but unfortunately that doesn't satisfy `>=1.8.6` which would break just about every plugin. --- src/package-lock.json | 2 +- src/package.json | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/src/package-lock.json b/src/package-lock.json index d1f16f8e7..51b2bd5a8 100644 --- a/src/package-lock.json +++ b/src/package-lock.json @@ -1,6 +1,6 @@ { "name": "ep_etherpad-lite", - "version": "1.8.16", + "version": "1.9.0", "lockfileVersion": 1, "requires": true, "dependencies": { diff --git a/src/package.json b/src/package.json index 1565939aa..284ec0ffb 100644 --- a/src/package.json +++ b/src/package.json @@ -248,6 +248,6 @@ "test": "mocha --timeout 120000 --recursive tests/backend/specs ../node_modules/ep_*/static/tests/backend/specs", "test-container": "mocha --timeout 5000 tests/container/specs/api" }, - "version": "1.8.16", + "version": "1.9.0", "license": "Apache-2.0" } From 0cc15df9b94f56fcd8444dfa7288b406c82d6813 Mon Sep 17 00:00:00 2001 From: Chocobozzz Date: Wed, 22 Dec 2021 17:33:13 +0100 Subject: [PATCH 119/446] Prevent pad translation and crash Prevent "TypeError: Cannot read properties of null (reading 'sheet')" exception because google chrome can translate `` title attribute --- src/templates/pad.html | 2 +- src/templates/timeslider.html | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/src/templates/pad.html b/src/templates/pad.html index 7ff447dc9..bc3cec88e 100644 --- a/src/templates/pad.html +++ b/src/templates/pad.html @@ -5,7 +5,7 @@ ; %> - + <% e.begin_block("htmlHead"); %> <% e.end_block(); %> diff --git a/src/templates/timeslider.html b/src/templates/timeslider.html index dc351b1d0..e26cd11e7 100644 --- a/src/templates/timeslider.html +++ b/src/templates/timeslider.html @@ -3,7 +3,7 @@ , langs = require("ep_etherpad-lite/node/hooks/i18n").availableLangs %> - + <%=settings.title%> Timeslider