Display Cabinet

Tag: full-stack

Rebuilding LWT’s API, part 2
Hi there! Much work has been done and the new LWT REST API is now ready!

For records, the initial suggestion, formatted with the help of ChatGPT, looked like the following (see part 1 for details):
```
1. GET API Endpoints:
   - Get API Version: `GET /api/version`
   - Get Next Word to Test: `GET /api/test/next-word`
   - Get Tomorrow's Tests Number: `GET /api/test/tomorrow`
   - Get Phonetic Reading: `GET /api/text/phonetic-reading`
   - Get Theming Path: `GET /api/text/theme-path`
   - Get Texts Statistics: `GET /api/text/statistics`
   - Get Media Paths: `GET /api/media/paths`
   - Get Example Sentences: `GET /api/sentences/{word}`
   - Get Imported Terms: `GET /api/terms/imported`
   - Get Similar Terms: `GET /api/terms/{term}/similar`
   - Get Term Translations: `GET /api/terms/{term}/translations`

2. POST API Endpoints:
   - Update Reading Position: `POST /api/reading/position`
   - Add/Update Translation: `POST /api/translation/{word}`
   - Increment/Decrement Term Status: `POST /api/terms/{term}/status`
   - Set Term Status: `POST /api/terms/{term}/status/set`
   - Test Regular Expression: `POST /api/regexp/test`
   - Set Term Annotation: `POST /api/terms/{term}/annotation`
   - Save Setting: `POST /api/settings`
```
It featured 11 endpoints on GET and 7 on POST. The key of this approach was it was not a fundamental change but rather a reorganization of the already existing AJAX requests. In fact, GPT do not have access to all the app information, as most endpoints are simply a simplification of the explanation sentence, and were sometimes optimistic or simply wrong: for instance GET /api/text/statistics, how are we supposed to know that we get statistics for a subset of texts here?

Finally, the chosen implementation is as follows:
```
1. GET API Endpoints:
   - Get Files Paths in Media folder: `GET /media-files`
   - Get Phonetic Reading: `GET /phonetic-reading`
   - Get Next Word to Review: `GET /review/next-word`
   - Get Tomorrow's Reviews Number: `GET /review/tomorrow-count`
   - Get Sentences containing Any Term: `GET /sentences-with-term`
   - Get Sentences containing Registred Term: `GET /sentences-with-term/{term-id}`
   - Get CSS Theme Path: `GET /settings/theme-path`
   - Get Terms similar to Another One: `GET /similar-terms`
   - Get Term Translations: `GET /terms/{term-id}/translations`
   - Get Imported Terms: `GET /terms/imported`
   - Get Texts Statistics: `GET /texts/statistics`
   - Get API Version: `GET /version`

2. POST API Endpoints:
   - Save Setting: `POST /settings`
   - Decrement Term Status: `POST /terms/{term-id}/status/down`
   - Increment Term Status: `POST /terms/{term-id}/status/up`
   - Set Term Status: `POST /terms/{term-id}/status/{new-status}`
   - Update Term Translation: `POST /terms/{term-id}/translations`
   - Create a New Term With its Translation: `POST /terms/new`
   - Set Text Annotation: `POST /texts/{text-id}/annotation`
   - Update Audio Position: `POST /texts/{text-id}/audio-position`
   - Update Reading Position: `POST /texts/{text-id}/reading-position`
```
We have 12 enpoints on GET and 9 on POST. Apart from the necessary corrections since GPT’s output, the new system clearly specifies when the server can use previous data or has to build it from scratch.

A previous “GPT” endpoint was GET /api/sentences/{word}, which would require a word to be passed as a text. Then the server would search all sentences containing this specific word, parsing and adapting it, which is expensive. When it is a new word, it is still done through GET /sentences-with-term, but for already registered words we use the word ID with GET /sentences-with-term/{term-id}, which is naturally translates as a SQL projection.

Another feat of this implementation is to remove the need to pass sensitive data such as SQL. When conducting a word review (test), only the server has knowledge of which subset of words should be tested, and shows one of these word to the user. Hence, the subset selection was stored as a part of the page URL. When a user would require the next word to test it would trigger a page refresh. Now, the URL only stores the ID of the language, text or words to test, as well as a unique ID for the test type (language, text or word). In this way SQL injection is prevented.

At the general scale, queries are now smaller and faster, as they support caching. The new architecture also allows to build tests with NPM, Chai and SuperTest, with 100% code coverage on GET.

Aftermath

The API is well integrated into LWT, the requests are smaller, the app faster and more secure. The API was released with LWT 2.9.0. Apart from a small fix in 2.9.1, the API had received no more fix (as of this time), proving its robustness. This RESTful system will now serve as a base to build a more dynamic LWT, paving the way to the future of the app.

As a first try building a RESTful API, I consider it a great success, and a great initiation to the art. Happy language learning!
2024-01-05
The Future of LWT-fork and LUTE

As announced in a GitHub post, I’ve had contributed enough to the current version of LWT to consider that it was time to roll over a new project, namely LUTE. In this blog I’m detailing the reasons for this change and it’s implications.

A Bit of History

After one year of contributing to my fork of LWT, I felt that I had done most of the reasonable housekeeping and small feature insertions. It was quite fun, but the more I worked, the more I felt that I had to stop contributing to the app at some point.

The original LWT stems from a great concept: learning by reading in your target language. However, the app had a chaotic development: it dates back to 2011, was made by a single person who is not a professional software developer, and never received contributions. The app expanded iteratively and was never intended to be production-ready. It has no test cases, no clean access to the database, etc.

Those were feature I struggled to import, but at some point I had to admit that there were too many things to do, and that a refactoring was no longer the solution. I started posting about a new app and discussing with the community, until a very motivated person came and saved the day: jzohrab.

He is an experienced software developer who decided to rebuild LWT from scratch at a time when I had my hands full fixing current LWT bugs. His intervention enabled me to pack up things with LWT while he was working on his software, enabling a smooth transition. As of May 2023, I consider both programs worth using.

LUTE + LWT-fork Era

LWT-fork is not a goner though. LUTE is a nice piece of software, but it hasn’t included all the features of LWT yet. More importantly, LUTE is jz’s software, and while we often collaborate together, our views can be diverging on some points. The organization we came into was that while he continues working on LUTE, I can freely solve problems on LWT, try new features, and import things I consider important to LUTE. With things as this, projects can benefit from each other, and users are free to use either of the two.

As a personal ambition for LWT, I now try to keep backward compatibility at a maximum while experimenting with new features that can be ported to LUTE, as stated in a previous discussion. LWT is still a nice playground for me because it deals with the basics of full-stack engineering (good to get experience), while LUTE relies on frameworks. Both are very nice to contribute to, and I hope you will find them useful.

If you liked any of these projects, please leave a star (LWT/LUTE stargazers) or fork them; that is our only reward as open-source developers!

2023-05-13
Maintaining LWT
Hi there!

In this post I’ll delve into a project which has taken a big part of my time for the last few months, the maintenance of an open-source language learning app: LWT.

Original LWT logo

LWT, standing for Learning With Texts, is a wonderful tool to learn text in your web browser, but it was quite abandoned 10 years ago, and it has not reached a larger audience since it was a bit difficult to use. So, I decided to blow the dust out of it, and started working on the GitHub community-maintained version. For those interested, the original project was posted at SourceForge.net.

Basically, the target was to give the tool the audience it deserved. The main targets were:
- Easy installation
- Mobile compatibility
- More secure
Of course, many subquests came along the way, and I finished by doing much more than expected. In the end, it was a good introduction to full-stack development and project management as many things were to be done in a very short time. Now, I think that the project is much easier to maintain, so that independent developers can figure out how to do their own stuff without having a hard time understanding the code.

Let’s talk a bit about the reason for which I started maintaining LWT. After I discovered LingQ, I found the tool very good, but also very expensive as many language learners are children. So, I decided to work on his open-source alter ego, LWT. Moreover, after spending time studying Japanese, I needed a robust tool to master the writing system, and reading books can turn out to be a pain if you cannot read words, let alone get the meaning.

Another motivation was a personal challenge. I mostly did front-side development, and this was a much more ambitious full-stack project. I had no previous experience in PHP, or more generally, how a server internally works, so it was a good introduction to the domain. I could also try a new set of tools and utilities, which is why in my first commits things were a bit confused.

Finally, I have been working on this project for almost a year now, and things are much better. I think I could tackle most of the urgent problems, and for the rest of the work, it will diverge from what LWT originally was and is much more related to where I want to go next. So stay tuned, as I may start on a successor for LWT from now on!
2022-09-17