More Import/Export/Sync

[xet7]

Some other recent issues, In Progress:

• Picture cannot be show #4803
• Try WeKan Snap Manual upgrade to newest Candidate? #4780
• Bug: Exporting big board timeouts #4603 Writing script to export big board JSON (not attachments necessary yet) before end of 2022-07
• Any files drag from file manager, email etc to attachments #2936 funding €1000 2022-07-26
• Shared templates #3313 at 2022-07-25 received €500 more funding
• Feature Request: Export card #4197 to PDF with all comments etc, will have postpaid €500
• Accessible drag-and-drop #459
• Test new WeKan with Meteor 2.7.2 #4512
• change langage not work #4518
• Wekan 6.30 - API login creates error 500 #4595
• Feature-Request: Number of cards per list and sum of custom number field in list head #3796
• The new added user that was assigned to existing team cannot be applied to kanban with the right authority. #4593
• Trying add existing user to a board - no result show  #4530
• Change time format with any language: YYYY-MM-DD / DD-MM-YYYY / MM-DD-YYYY
• Separate migration app for copying attachments from MongoDB 3.x/4.x/5.x directly to S3 etc
• Feature-Request: PI Program Board with Red Strings #3392
• More at Roadmap

More Import/Export/Sync

I'm currently at EU Dapsi 3nd CALL with project name Multiverse. After 1st phase, I did yesterday give 10min talk about 1st phase results, where was this slide 3:

Solutions to importing data to WeKan, being integrated:

• Trello attachments: https://github.com/wekan/wekan/tree/master/trello
• Jira: https://github.com/wekan/wekan/wiki/Jira
• Asana: https://github.com/wekan/wekan/wiki/Asana
• Zenkit: https://github.com/wekan/wekan/discussion
• CSV: https://github.com/wekan/wekan/wiki/CSV

After that talk, only question was, that why there is only these usual ways to import. There should be more, to make WeKan more useful. So, in 2nd phase (if at my 1st phase evaluation EU will decide to accept results and allow me to continue to 2nd phase), my plan is to add these and more to WeKan. Sure other goals mentioned in those slides are also nice to have, but this Import/Export/Sync is the most important.

[xet7]

Someone mentioned to me, that exporting small boards works, but not exporting big boards.

[xet7]

Also there is some issue somewhere, that exporting board does not export attachments.

[xet7]

I was accepted to EU project 2nd phase of my Multiverse project.

Question is, to what direction would WeKan community like to go ? Multiverse direction, Meteor direction, or both? Here are details what is ahead on each direction.

a) Multiverse

1. Minimize frontend to HTML4/CSS, without Javascript, so it works on all browsers, like these:

• Netsurf (Windows, Linux, RISC OS, ReactOS, Redox OS, etc)
• Amiga AWeb
• FreeDOS Dillo

Minimized frontend by default will shows only small part of board, and buttons to view other parts of board. Maybe map of board parts, possibility to click that map part.

2. Add accessibility tags etc.

3. After frontend is minimized and works without Javascript, add back some Javascript features with minimal code, like drag-drop, etc.

4. Add back backend, like some parts of Meteor WeKan, or CloudFlare/AWS/Azure/Google Serverless. Point is, when frontend is minimal, it loads very fast, and most processing happens at backend.

5. Add support for other databases, with some DAL/ORM, or per-DB separate queries. Having SQL database support will make implementing Feature-Request: Number of cards per list and sum of custom number field in list head #3796 much easier. Also, for All Boards view, I tried to make some Javascript MongoDB query to get first characters of board titles with count amounts of boards query, like I many years made for some library database, but I did not get it working with MongoDB aggregate query yet, it's much more difficult syntax than SQL, and MongoDB query also had some problems with UTF-8. I would presume that SQL queries with UTF-8 would work much better with SQLite and for example Go or Tcl.

Multiverse is currently In Progress here:

• Convert MongoDB to SQLite
• PHP prototype. PHP is in this prototype only during minimization, after that some other programming language could be used for backend.

Also, in Multiverse project https://dapsi.ngi.eu/hall-of-fame/multiverse/ the plan is "WeKan multiple Import/Export/Sync to UI Designer". This means, that when frontend is minimized, WeKan UI will be made much more configurable, more settings, for being able to clone any app much more easily. This means more Import/Export/Sync options, ability to change font, background images, positions of menus, colors, etc at many more places.

These all speedups are In Progress because of EU project and some customers asking for more scalability, like to use WeKan with 22 000 users. Having enough progress with Multiverse would require flexibility that too many customers would not be so strict about deadlines. Multiverse is what I'm In Progress of doing currently, because I need to give presentation about results in upcoming EU 5min talk.

b) Meteor

1. Current Meteor limits are, that if both client and server computers have 32 GB RAM, WeKan uses a lot of CPU and RAM both at client and server.

2. Client CPU usage is because there is about 10 MB of Javascript loaded at WeKan page. Running that amount of Javascript requires fast desktop computer and fast browser like some Chromium based browser.

3. WeKan loads all cards of board to browserside minimongo database, that is made with Javascript. This can cause browsers to crash because of too much data. Loading of cards is made with PubSub. Meteor reads all newest changes from MongoDB and immediately updates it to all users boards. Amount of loaded data could be limited, if with someone would add code it would be detected what swimlanes are visible, and only for those start Publications/Subscriptions at wekan/server/publications/, and stop those that are not visible.

4. Serverside WeKan currently has growing CPU and RAM usage, because of memory leaks. This could be debugged with:

• https://forums.meteor.com/t/meteor-v2-8-memory-leak-in-mongo-driver/59101/2
• https://forums.meteor.com/t/solved-meteor-server-memory-leaks-profiling/58210/21
• https://blogs.windows.com/msedgedev/2021/12/09/debug-memory-leaks-detached-elements-tool-devtools/
• https://engineering.fb.com/2022/09/12/open-source/memlab/
• https://www.infoq.com/news/2022/09/memlab-javascript-memory-leaks/

5. For MongoDB database drivers, only MongoDB driver for Golang still supports newest and oldest MongoDB versions that have been used in various versions of WeKan. Other drivers have dropped support for older versions of WeKan. Trying to migrate big MongoDB databases like 500 GB size does take about 4 hours, and also depends on is there enough disk space available. MongoDB database content moved to SQLite results in much smaller file size. If staying with MongoDB, first would be migration from Snap MongoDB 3.2.2 to MongoDB 5.x. And newest version of MongoDB is 6.x, that brings some encrypted storage and search options. Meteor has not yet added support for MongoDB 6.x. Meteor has some plans to add support for other databases Question: back end other than Mongo? #787 (comment) . For attachments, there is Meteor WeKan features to move those to filesystem. Still supporting all versions of MongoDB would require using somehow both newer and older Node.js drivers, or using some Go migration tool, or writing some kind of Node-MongoDB driver with Go. Newest Meteor only supports MongoDB 5.x, in database query syntax etc.

6. MongoDB 5.x requires AVX instructions Latest MongoDB images require AVX instructions - Debian VMs fail to run #4321 , so to have WeKan working on some hardware, it requires changing to some other database.

7. MongoDB has changed to SSPL license, some don't like it:

• https://github.com/mongodb/mongo/blob/master/LICENSE-Community.txt
• https://en.wikipedia.org/wiki/Server_Side_Public_License

c) Both Multiverse and Meteor

There are some genius contributors that have been adding new features to Meteor all the time. It could be waited until some Multiverse WeKan parts can be changed to Meteor WeKan, for in-place speedups.

[xet7]

Related to Multiverse mimization, I'm part of "One Week In DOS challenge":

https://lunduke.substack.com/p/the-live-in-dos-for-a-week-challenge

These are my goals:

a) WeKan Open Source kanban working on FreeDOS Dillo webbrowser. Current server code is at https://github.com/wekan/php at public/index.php (yes one file, not like Meteor WeKan 60 000 files)

b) Try some DOS programming languages to create WeKan webserver.

c) Try to create TUI with code based on https://github.com/xet7/darkesthour

d) Try TRSE (Turbo Rascal Syntax Error) does it work for DOS etc, is it possible to create WeKan GUI

e) Porting MongoDB to SQLite, and using SQLite on DOS. Current In Progress code is at https://github.com/wekan/hx/tree/main/prototypes/database/MongoDB/mongoexport-csv-sqlite . Current Meteor WeKan uses MongoDB that does not run on DOS.

f) After WeKan is minimized to DOS, it will also run faster on versions for web, serverless, etc.

When doing DOS challenge, I'm also thinking how to implement native apps #480 (comment)

[xet7]

Related to DOS challenge, here is Multiverse prototype in Dillo webbrowser in FreeDOS:

[xet7]

Related to UTF-8 and Emoji, here is newest standard:

• https://blog.unicode.org/2022/09/announcing-unicode-standard-version-150.html
• https://lwn.net/Articles/908032/

[xet7]

MongoDB to SQLite conversion progress

For most text data I have now conversion script to sqlite at https://github.com/wekan/hx/tree/main/prototypes/database/MongoDB/mongoexport-csv-sqlite.

Later I will add also exporting attachments from MongoDB GridFS to files. Because some attachment filenames are same (duplicates), I will:

1. Make some SQL queries to find correct fileIDs, because fileIDs are unique
2. Make script, that for each found fileID save file to separate file with mongofiles https://github.com/wekan/hx/tree/main/prototypes/database/MongoDB/mongofiles
3. Cleanup filenames, like urlencode/urldecode etc, if needed

DOS Challenge progress

FreeDOS seems to have SQLite, but without Tcl. Trying to find some programming language for DOS that has SQLite included, or compile something.

[ocdtrekkie]

/sub for updates. A non-Meteor version would resolve a lot of the pain the Sandstorm version has fought with Meteor and Mongo too.

[xet7]

I found some new vanilla js drag drop library, that I have not yet tested, could be faster:

• https://www.dflex.dev
• https://github.com/dflex-js/dflex
• https://news.ycombinator.com/item?id=32852020

I'll also try to find other minimal code.

[xet7]

For each input field, to get them visible on HTML4/CSS only, I'll try to convert all these at serverside to HTML4 text and bitmap images like PNG/JPG:

• SVG (that could also have some unwanted Javascript, it would not show in bitmap image at all) to PNG
• Markdown to HTML4
• Mermaid to SVG to PNG
• Emoji to OpenMoji images that are currently here at PHP prototype. For ALT text and accessibility, add tags that explain what emoji is about.
• Also for any image, if there is some description, add ALT or other accessibility tag

For saving data:

• When some text is saved to textfield or modified, maybe also convert it to HTML4/images/etc, for faster viewing

For Import/Export:

• Try to add to Import/Export file formats also SQLite, because SQLite files are much smaller than JSON

[xet7]

DOS Challenge part 2: About coding without dependencies, if possible

In discussion about Making software old school style:

From @xet7

Maybe kind of related. Current Meteor WeKan features are done like adding more npm dependencies. Instead, new Multiverse WeKan I try to do by writing those features myself with minimal code, without adding any dependencies, if possible.

From ArchieT

I like a good kanban board. and NOW I'm interested in taking a look at your new version. I think I was quickly turned off by the earlier version for just the reasons you mention.

From @xet7

Thanks a lot, this really means a lot to me. I really need a lot of encouragement to continue.

[xet7]

DOS Challenge part 3: Next steps for prototypes

• Create repo for WeKan for DOS: https://github.com/wekan/wedos
• PHP prototype: divide index.php back to separate files, so that it's not all of 1000+ lines of code in same file, would open faster to ext editor
• Main menu: .bat files that use SQLite. Kind of inspired by https://secretgeek.net/dod_intro at https://github.com/secretGeek/dod , but not using it's code.
• Menu option: Create new database. For it, save database structure from current wekan.sqlite to .bat file that creates new blank database.
• DOS prototype: Convert PHP files to .bat files that uses SQL queries to show board etc.
• Javascript prototype: Convert PHP/.bat based scripts to Javascript, to use in WeKan
• Javascript prototype: Copy Javascript files from Meteor WeKan to separate Javascript files by feature, to logical directory structure
• Try does each Javascript feature work with:
  • DOS https://github.com/SuperIlu/DOjS
  • Newest Node.js 18.x
  • CloudFlare Workers
  • Azure Serverless
  • AWS Serverless
  • Google Cloud Serverless
  • Is there Javascript for Amiga?

[xet7]

DOS Challenge part 4: WeKan add multiple Import/Export/Sync to UI Designer, making it possible to create any app

From EU Dapsi, about WeKan Multiverse

We would like to inform you that the final technical evaluation will start on Monday, November 1st 2022. Therefore, we kindly ask you to send your final deliverables/prototypes latest by October 31st 2022.

In the second phase, we examine and evaluate mostly your implementation and the running prototype. Therefore, we are expecting:

• A descriptive video showing and explaining the running prototype
• Document (Max. 5 pages) describing the prototype and test results, performance and best practice (if there is any), etc.
• Link to the code repo (e.g., Github)
• Accessible prototype (e.g., website) is not mandatory but will be considered as a strong point

Progress of some parts:

• Talk is about 5 slides, 5 min, non-technical. Recording at 5min pitch today 15.20 Paris time about Multiverse WeKan #4722 (below is more techical points)
• MongoDB text data to SQLite https://github.com/wekan/hx/blob/main/prototypes/database/MongoDB/mongoexport-csv-sqlite/mongoexport.sh
• IN PROGRESS: MongoDB attachments to files to S3 etc
  • Make SQL queries what is file ID
  • Use this code to save that file ID from MongoDB to file https://github.com/wekan/hx/blob/main/prototypes/database/MongoDB/mongofiles/get-id.sh
  • Create MD5/SHA256/etc hashes
  • Deduplicate: search does that file hash already exist in SQLite database
  • Create thumbnails of JPG/PNG etc images
  • Maybe detect text from images, for fast text search?
  • Create text versions of documents with Pandoc or similar, for fast text search
  • From PDF/DOC/etc maybe create SVG or JPG, for viewing those files in browser
  • From videos maybe create some thumbnails with FFMPEG ?
  • From videos maybe create smaller version with FFMPEG ?
  • Encrypt files with RClone or something else?
  • Save encrypted file hash to https://databunker.org repo at https://github.com/securitybunker/databunker
  • Upload files to S3 or other storage with Meteor-Files or RClone https://rclone.org repo https://github.com/rclone/rclone
• Files
• Architecture diagram paper to send in email to EU Dapsi at end of 2022-10
• Working demo of Multiverse WeKan
• GDPR related additions?
• Maybe optional encryption:
  • At frontend with Javascript or native app
  • At backend code
  • At database, like MongoDB or SQLite encryption
  • How fast is encryption/decryption? What about E2E encryption, how did RocketChat do it, or how did Nextcloud improve speed of E2E ?
• Scalable? How much more speed can be added? How many request per second? How small amount of HTML4/CSS and little part of board visible?
• Are you happy with framework used?
• Are all dependencies up-to-date and maintained?
• Import/Export/Sync integrated:
  • Trello
  • Jira
  • Asana
  • Zendesk
  • Others from WeKan GitHub issues, and much more non-usual ones
  • Think how to import any kind of JSON/CSV etc structure (also nested structure), with mapping of fields, or adding new fields. Maybe import/export/sync wizard for any file format?
• UI Designer
  • More WeKan hardcoded parts to be configurable
  • More IFTTT Rules
  • Convert all themes to settings page: Is card border round or square, custom color, etc
  • Modifying more what is visible or not: Menus, buttons, etc
  • Think how to make anything editable, with translatable settings menus etc
  • Think how to add some custom UI page to WeKan
  • HTML4/CSS, works without Javascript. If optionally Javascript enabled, then additional features available.

Related links:

• https://dapsi.ngi.eu/20-new-solutions-to-bring-the-power-of-data-back-to-peoples-hands/
• https://dapsi.ngi.eu/hall-of-fame/multiverse/

Screenshot about what is should look like

[xet7]

DOS Challenge part 5: Group and count by first character of board names

When there is a lot of boards, it is very useful to group by first character of board (and after that click charater to see list of boards for that character, to be added later). I could not figure out how to do it with MongoDB aggregate query. Here is how I did it with SQLite, for type board. SQLite returns results immediately. Note that emojis, UTF-8 shows also on Netsurf browser. TODO is to figure out, could I get these visible also in FreeDOS Dillo browser.

SELECT _id, substr(title,1,1), COUNT(*), type from boards
WHERE type=:type GROUP BY substr(title,1,1) ORDER BY substr(title,1,1) ASC;

https://github.com/wekan/php/blob/main/page/allboardschar.php

[xet7]

Related my 5min pitch talk video today #4722

[xet7]

Meteor 2.8 memory leak in MongoDB driver?

• Meteor v2.8 - Memory leak in mongo driver? meteor/meteor#12321
• https://forums.meteor.com/t/meteor-v2-8-memory-leak-in-mongo-driver/59101/2
• https://jira.mongodb.org/browse/NODE-4833

[razum2um]

Going Multiverse route would kill the most precious feature: reactive updates and realtime collaboration (i.e. when team does a sprint planning) - which's finally done not using screenshare. It's like requesting https://etherpad.org/ to run without JS

What is the end goal of Multiverse route?

[xet7]

@razum2um

Going Multiverse route would kill the most precious feature: reactive updates and realtime collaboration (i.e. when team does a sprint planning) - which's finally done not using screenshare. It's like requesting https://etherpad.org/ to run without JS

No, it would not. If you have JS enabled in browser, reactive updates and realtime collaboration will still work. And even without JS, you would still see newest changes after any click.

Also, Multiverse means multiple versions of WeKan. It's not about killing any of those versions, or about killing any feature. It is about keeping all features, and adding more features.

AFAIK @mfilser is currently working to make additional speed improvements to Meteor WeKan.

Moved to here from #4321 (comment)

what are real minimum wekan's mongodb requirements? AFAIK, there's nothing what forces (will force) to upgrade to 5x, right?

Meteor web framework tries to use newest MongoDB version and MongoDB driver to get security updates etc. Currently Meteor is compatible with Node.js 14.x and MongoDB 5.x . Having MongoDB 5.x support did mean code changes to both Meteor and WeKan, database query syntax is different, because MongoDB changed index syntax from _ensureIndex to createIndex. In MongoDB 6.x MongoDB has deprecated more MongoDB syntax, so that also did mean changes to Meteor code (that Meteor devs made). Newest Meteor beta has support for MongoDB 6.x , that is included in upcoming version of WeKan.

WeKan at https://sandstorm.io is still stuck with MongoDB 3.0, so that means I try to find some way to use previous Node.js MongoDB driver that still supports MongoDB 3.0. Or alternatively, have some migration from MongoDB to SQLite.

besides, somebody could care about mongodb's license. The last versions licensed as AGPL version 3 are 4.0.3 (stable) and 4.1.4

Yes, that's why I'm trying to add support for SQLite.

tbh, I don't understand a lot of things happening

• why should we upgrade mongodb and believe the new version is better

Newest WeKan is made with newest Meteor web framework, that is currently only compatible with MongoDB 5.x

• why 22K users have to fit inside 1 server

Well, it's 30K. No they don't currently fit, they are using Kubernetes with maybe 18 pods.

• why SaaS are trying to serve the whole world in 1 central place (in contrast how mastodon works)

Maybe some other SaaS is trying, but not WeKan. Most self-host WeKan, using some WeKan Platform like Snap/Docker/Sandstorm/Kubernetes/UCS/SourceBundle etc. I only have my limited demo instance, and handful of customers I host.

• why not to stay a perfect thing for small-mid size

Hmm? It's not like there would be any feature removed from WeKan.

and support server-to-server interactions

What server-to-server interactions?

What is the end goal of Multiverse route?

• Minimize amount of frontend code, so pages load fast. Load only visible area.
• Be fast enough for those that have 30k users running Kubernetes on high latency wide area networks
• Scale down, like with C and SQLite or textfiles, to work on Amiga that has 2 or 18 MB RAM. Related https://github.com/xet7/darkesthour
• Scale up, like CloudFlare Workers with SQLite, wordwilde low latency, and other planets
• Work without websockets (Meteor requires websockets), on some corporate networks websockets are not allowed or cause a lot of problems
• Work without reactive/realtime updates, because with when there is 30k users at network, it needs minimal amount of network traffic to work at all. Current WeKan has about 10 MB of Javascript at frontend, that's too much. If someone is clicking buttons or saving data at small part of board, that someone will see newest updates after each click, because it does full reload of that small minimal amount of part of board that is currently visible.
• Work with all browsers, like Netsurf etc, that do not have Javascript support.
• Have UI Designer, or a way to more freely modify WeKan UI.

[xet7]

About designing for speed some discussion at #4823 (comment)

[xet7]

About using SQLite as graph database:

• https://github.com/dpapathanasiou/simple-graph
• https://www.hytradboi.com/2022/simple-graph-sqlite-as-probably-the-only-graph-database-youll-ever-need
• https://news.ycombinator.com/item?id=25544397

[xet7]

About near future #3948 (comment)

[xet7]

I did get working scripts to move WeKan MongoDB GridFS files to Minio, and MongoDB text to SQLite:
https://github.com/wekan/minio-metadata

Next I will do:

• Small refinements to those Minio scripts
• Continue coding feature to WeKan to view and use Minio files

[xet7]

About upcoming Meteor 3.0 #4851 (comment)

[xet7]

It's now possible to build WeKan with Meteor 2.11 directly on arm64 like Asahi Ubuntu M1/M2, OrangePi, RasPi4:

meteor/meteor#12160 (comment)

[xet7]

News 2023-04-12

FerretDB

More about WeKan with FerretDB/PostgreSQL/SQLite instead of MongoDB Progress:
#787 (comment)

Node.js 14

Meteor Announcing Extended Support Maintenance for Node.js 14 for one year. This gives Meteor developers one year more time to upgrades, with getting security patch backports to Node.js 14 from Meteor, helping Meteor developers sleep better. This is very helpful, because official EOL for Node.js 14 is in 2 weeks https://endoflife.date/nodejs .
https://blog.meteor.com/announcing-extended-support-maintenance-for-node-js-14-f9e8381f8bb5 . Node.js 14 ESM repo is at https://github.com/meteor/node-v14-esm

[xet7]

Moved to here from #4893 and #4895

Speed

Having some lag in Snap Stable is not what could be thinked to be final speed.

These upcoming WeKan upgrades will affect speed:

1. Upgrade to Snap Stable to have fixes included from Snap Candidate when upgrading works and is released, maybe this or next month.
2. Upgrade to newest Meteor 3.0 that has Node.js 18 and newest MongoDB and many speed improvements to Meteor, this year.
3. Pull request sometimes coming from @mfilser that optimizes database queries to return less data, maybe this or next year.
4. Moving attachments from MongoDB to Minio https://github.com/wekan/minio-metadata , maybe this or next month.
5. Having possibility to use PostgreSQL or SQLite with FerretDB, see above comment More Import/Export/Sync #4578 (comment) , maybe late this year or next year.

What you can look if you have problem with speed:

• Do you have some script using WeKan API, like inserting card, all the time without any delays between API calls? It could be useful to have delay of about 7 seconds between each API call.
• Are you running WeKan and MongoDB using SSD or NVME disk? HDD disk is very slow. When opening big board, speed difference is HDD 5 minutes vs SSD 2 seconds.
• Do you have Webhooks that do not get immediate reply of 200 ? Or Webhook URL that does not work? That slows down WeKan. To fix it, look at MongoDB database integrations table/collection , remove content from there, with MongoDB GUI like DBGate or NoSQLBooster https://github.com/wekan/wekan/wiki/Backup#dbgate-open-source-mongodb-gui
• Do you have Global Webhooks enabled at Admin Panel? That sends all changes to webhook URL. If webhook receiver is slow, it slows down WeKan.

[xet7]

Meteor 3.0 progress from RocketChat developers

I read from Meteor Slack, that RocketChat developers had made async await related code changes, and now are doing fixes to Meteor to get Meteor 3.0 fully working. While those changes are not yet merged to Meteor, RocketChat developers have their own fork of Meteor where there are making progress:

https://github.com/RocketChat/meteor

[xet7]

About recent updated top dependencies: jquery, summernote WYSIWYG editor, bootstrap

#4511 (comment)

[xet7]

https://github.com/wekan/wekan/wiki/Deep-Dive-Into-WeKan

https://github.com/wekan/wekan/wiki/WeKan-Multiverse-Roadmap

[xet7]

Hi,
I just got news, that FerretDB 1.18 now has support of OpLog. So I will be trying replace MongoDB 6.x with FerretDB/PostgreSQL or FerretDB/SQLite.

https://github.com/FerretDB/FerretDB/releases/tag/v1.18.0

FerretDB/FerretDB#1752 (comment)

https://forums.meteor.com/t/ferretdb-1-18-now-has-oplog-support-trying-replace-mongodb-6-x-with-ferretdb-postgresql-or-ferretdb-sqlite/61092

meteor/meteor#12150 (comment)

Related #5223 #4321 #787