Version 3.3.0 · All versions

These are the docs as released with Solid Memo 3.3.0. Read the latest docs.

Format versions and migrations

The way Solid Memo stores data changes as the app grows, and a Pod may hold data an older version wrote. Every stored item says which format version it is in. This page says how the app brings old data up to the current format, leaves data from a newer version alone, and updates a deck copied from the library to its newer release.

How Solid Memo tells old data from new, and how it brings a pod up to the format the current app writes. The formats are the shapes; the steps between them are the migration modules in domain/shapes/migrations; the plan is in domain/migration.ts; the notice the user sees in MigrationContainer.tsx.

Versions

Every subject Solid Memo writes carries sm:formatVersion (a missing one means 1, the format that predates the field). LATEST_VERSION in the generated shapes module is what the app writes today; DECK_FORMAT_VERSION and friends are aliases of it.

Class12Why the version moved
Instancetitle, createddcterms:replaces (the instance it is an updated copy of) and dcterms:modified (when it replaced it), both optionalThe format update of that time wrote a copy and kept the original as a backup; the copy records which one. The update now writes each document where it is and states neither; an instance an earlier update made keeps them, and the app ignores them (The backup).
Decktitle, document links, provenancesm:direction, statedA format-1 reader would study a bidirectional deck one way only and count the other way's review subjects against the day's budgets without matching them to any card.
Cardsm:front and sm:back, both requireda side may be a picture (sm:frontImage / sm:backImage, an IRI), text, or bothA format-1 reader treats a card without sm:front as malformed and drops it, so picture-only cards must not be mistaken for format 1.
Review statethe SM-2 fields; a snapshot and per-direction subjects were added without a bump, so format 1 admits themthe same fields; the snapshot is all five triples or none; the subject naming (#<cardId>, #<cardId>@back-to-front) is part of the contractStamping begins: a format-1 reader meeting a format-2 state would silently ignore the snapshot and the other direction, which is what a version is meant to flag.
Preferenceswhatever the unversioned era wrote: every field optional, a missing one meaning its defaultevery field statedThe document is rewritten on every save, so stamping costs nothing; format 2 is the complete record.

Deck format 4 (and library deck series format 2, its summary in the library index) states a deck's title and description as language-tagged text: one value per language, exactly one of them English, which the app shows and edits while keeping the other languages. The version moved because a format-3 reader knows only untagged strings: it would read a format-4 deck as having no title and drop it. The step to format 4 tags a format-3 deck's title and description as English.

Card format 3 adds a note under each side (sm:frontNote, sm:backNote, shown once the answer is revealed) and a label above the back (sm:backLabel, how the answer relates to the front), all three language-tagged text like a deck's title, and lets a card be retired (owl:deprecated true): a library deck keeps a card it no longer uses instead of removing it, so a copy keeps the card and its review state but no longer studies it. The version moved because a format-2 reader would go on studying a retired card. A format-2 card is in use, so the step to format 3 changes nothing.

Card format 4 lets a side's text be language-tagged, one text per language ("Mona Lisa"@en), or stay untagged when its language is not known, never both. English is not required, since card text is often not English. The app shows the reader's language, else the English, else the untagged text, and edits one text while keeping the others; text typed in the app is untagged. The version moved because a format-3 reader knows only untagged text: it would read a tagged side as empty and drop the card. The step to format 4 changes nothing in the data, as a format-3 side's text is untagged and stays so: no language is guessed.

Deck format 5 and card format 5 let a deck's title and description, and a card's notes and label, be language-tagged text in any language, one value per language: English is no longer required. A deck titled only in Swedish, or a note only in Finnish, is whole. zxx tags text in no language (codes, numbers, symbols). Library releases stay at deck format 4: English first is the library's curation policy, not a rule of the data, so no release is rebuilt; an imported release is written in format 5. The version moved because a format-4 validator flags a title or note with no English, and a format-4 app drops such a note when the card is edited. The steps to format 5 change nothing but the version (restamped: nothing guessed): every format-4 deck and card is valid format 5 already. Text the step to deck format 4 tagged English and text saved the same in English and another language stay as they are: a pure step cannot tell an identical English copy from a real translation ("Stockholm"@en and "Stockholm"@sv), and the app accepts the same words in several languages as text in each of them, so the format-4 era's identical English copies are left in the data untouched. An untagged card side stays untagged.

Deck format 6, with library deck format 5 and library deck series format 3, lets a deck's keywords (dcat:keyword) be language-tagged, several per language ("capitals"@en, "huvudstäder"@sv), so the app can show a deck's keywords in the reader's language. The version moved because a format-5 reader knows only untagged keywords: it would read a tagged keyword as absent and drop it on its next save of the deck. The steps to deck 6, library deck 5 and series 3 keep untagged keywords untagged, their language unknown ("" in the model): no language is guessed. Untagged keywords are valid only as kept from older formats: the app writes every new keyword under the language the user states. Library releases from library deck format 5 on tag every keyword (each deck has keywords in English and Swedish); older releases stay frozen with untagged keywords, and an import of one writes them untagged into the format-6 copy.

Card format 6 (vocabulary 1.20) gives the front a label above it (sm:frontLabel), as format 3 gave the back one, so a card studied both ways says what kind of thing is asked ("Country" above "Croatia", "Capital" above "Zagreb"). The version moved because card format 5 was published without the term, and a published shape is never edited (vocab.md). A format-5 card has no front label, so the step to format 6 changes nothing but the version (card/5-to-6). Library releases stay at the card format they were published in; a release with front labels is written at card format 6, and an import writes every card at 6. An app from before format 6 reads a format-6 card with the format-5 shape, without its front label, and does not write over it (Versions), so updating stays the user's choice.

Deck groups (deck group format 1) need no migration: groups are new subjects, and a deck's sm:position and the catalog's dcat:catalog belong to no shape, so the deck and catalog formats are unchanged.

Courses (vocabulary 1.14) need no migration and moved no format version:

An older app that edits a course's card keeps its sm:distractor links, as it keeps any triple its shape does not own. One that removes the card leaves its sm:Distractor subjects behind, which this app ignores, since no card names them. An older app's check lists an sm:Distractor as untyped, a subject of no class it knows: visible, never a violation.

Text formats (vocabulary 1.15) need no migration and moved no format version. sm:textFormat joined card format 5, step format 1 and chapter format 1, optional. This is a judgment call, recorded here with its reasoning:

What a review state is of, and the other links of vocabulary 1.16 (see vocab.md), need no migration and moved no format version:

Library deck format 6 (vocabulary 1.18) lets a release describe its series and its publisher in its own document, so a release published in a pod is whole without an index (deck-library.md):

What an answer says of how it was given in vocabulary 1.19 (response times, the day the prompt was due and the device's offset from UTC) needs no migration and moved no format version:

Rules that hold across versions:

The chain

The migration chain
Each shape's record versions in order, with the migration that takes one version to the next.
The diagram's source
flowchart LR
    accTitle: The migration chain
    accDescr: Each shape's record versions in order, with the migration that takes one version to the next.
    v1["CardV1"] -->|card/1-to-2| v2["CardV2"]
    v2 -->|card/2-to-3<br/>in use: nothing to change| v3["CardV3"]
    v3 -->|card/3-to-4<br/>text stays untagged| v4["CardV4"]
    v4 -->|card/4-to-5<br/>restamped: nothing guessed| v5["CardV5"]
    v5 -->|card/5-to-6<br/>no front label yet: nothing to change| v6["CardV6"]
    d1["DeckV1"] -->|deck/1-to-2<br/>direction: front-to-back| d2["DeckV2"]
    d2 -->|deck/2-to-3<br/>DCAT dataset: direction concept,<br/>default description, creator agents| d3["DeckV3"]
    d3 -->|deck/3-to-4<br/>title and description tagged English| d4["DeckV4"]
    d4 -->|deck/4-to-5<br/>restamped: nothing guessed| d5["DeckV5"]
    d5 -->|deck/5-to-6<br/>keywords: untagged kept| d6["DeckV6"]
    l2["LibraryDeckV2"] -->|libraryDeck/2-to-3<br/>release 1 of its series| l3["LibraryDeckV3"]
    l3 -->|libraryDeck/3-to-4<br/>title and description tagged English| l4["LibraryDeckV4"]
    l4 -->|libraryDeck/4-to-5<br/>keywords: untagged kept| l5["LibraryDeckV5"]
    l5 -->|libraryDeck/5-to-6<br/>nothing to change| l6["LibraryDeckV6"]
    s1["LibraryDeckSeriesV1"] -->|libraryDeckSeries/1-to-2<br/>title and description tagged English| s2["LibraryDeckSeriesV2"]
    s2 -->|libraryDeckSeries/2-to-3<br/>keywords: untagged kept| s3["LibraryDeckSeriesV3"]
    r1["ReviewStateV1"] -->|reviewState/1-to-2<br/>partial snapshot dropped| r2["ReviewStateV2"]
    p1["PreferencesV1"] -->|preferences/1-to-2<br/>defaults filled| p2["PreferencesV2"]
    p2 -->|preferences/2-to-3<br/>the default invalid data policy:<br/>set invalid data aside| p3["PreferencesV3"]
    p3 -->|preferences/3-to-4<br/>theme as the browser prefers| p4["PreferencesV4"]

One module per step (<class>/<n>-to-<n+1>.ts), each a pure, total function from the record of one version to the record of the next, never mutating its input. Besides the record a step is given the IRI of the subject being migrated (MigrationContext), for a step that names new resources beside it. migrate(shape, record, { subject }) walks the chain to the latest version; a gap is a programming error and throws. Tests assert that the chain is contiguous for every kind, that every step is pure, that the latest record passes through untouched, and — with the real shapes — that every step's output conforms to the shape it moves to.

The pod migration

The format update brings each outdated document up to this app's formats on its own, where it is, in one write the pod makes whole or not at all. No document, subject or folder changes its address, no registration in the type index changes (only those missing are added), and nothing is kept beside the documents: no copy, no backup. A document that cannot be updated now is left as it is, readable, and the update goes on with the others; run again, it updates what is still outdated (instanceUpdate.ts, updateInstance in useCases.ts).

Updating an instance
Once the learner confirms, the update reads what is outdated, writes one document after another, and registers what the type index lacks once nothing failed.
The diagram's source
flowchart TD
    accTitle: Updating an instance
    accDescr: Once the learner confirms, the update reads what is outdated, writes one document after another, and registers what the type index lacks once nothing failed.
    open["Instance opened"] --> plan["planMigration<br/>read meta, preferences,<br/>every deck's cards and reviews"]
    plan -->|nothing outdated| quiet["(nothing shown)"]
    plan -->|outdated| notice["Notice: what will change"]
    notice -->|Update…| confirm["Confirm: how the update<br/>keeps the data safe"]
    confirm -->|Start the update| read["1 read: what is outdated now,<br/>planned again"]
    read --> write["2 write, one document after another:<br/>read it, bring its outdated subjects up to date,<br/>write it in one edit, If-Match the read"]
    write -->|written| updated["Updated"]
    write -->|refused or failed| left["Left as it is,<br/>named with why"]
    updated & left -->|the next document| write
    updated & left -->|none left| register["3 register: once nothing failed,<br/>the registrations missing<br/>from the type index"]
    register --> done["Done; or what is left, and why:<br/>Try again updates only that"]

On a server that ignores If-Match (node-solid-server), a write cannot be held to the version read: a change another device makes between the update's read of a document and its write is kept by a PATCH, which changes only the update's triples (and fails when a triple it deletes is gone), but undone by the PUT of a whole large document, as by any such save there (data-model.md).

What a server cannot apply leaves that document as it was, as any failure does: node-solid-server's parser knows no SPARQL-style PREFIX directive, so it cannot patch a document another app wrote with one, nor delete by a PATCH a value the update rewrites spelled otherwise than Solid Memo spells it (2.50 for 2.5).

Proof on a real server

npm run test:pod runs the update — the app's own use cases and Solid adapters, wired as in createAppUseCases — against real Solid servers (each the tests start: two majors each of the Community Solid Server and node-solid-server), recording every HTTP request as the app attempts it and as it reaches the server (instanceUpdate.integration.test.ts). It seeds an old-format instance with another app's triple, an unknown file and shared access, and checks that:

On Community Solid Server 6, whose ETags are to the second, the update's test of a change made "elsewhere" waits for the next second before it makes it; on node-solid-server, which ignores If-Match, the tests of such a change are skipped (testing.md). The tests start a Community Solid Server in memory themselves; set SOLID_SERVER_URL to use another server that lets anyone read and write. npm test does not run them.

The backup

The format update keeps no backup: what it writes never needs putting back, as each document is in a format the app reads, updated or not. What a backup would still guard against is a step that runs but loses meaning; the chain's tests guard against that (The chain: each step pure, its output conforming to the shape it moves to), as for every save, which applies the same steps with no backup.

Until stable addresses, the format update copied the whole instance into a sibling folder (<name>-<uuid>/), updated the copy, and kept the original as a backup, which the copy's meta.ttl names (dcterms:replaces). Such a backup, or a partial copy such an update's closed tab left, is now an ordinary folder that the app no longer lists or offers to restore, which the user may delete by hand; meta.ttl keeps dcterms:replaces and dcterms:modified, which the app ignores.

Repairs edit in place, as the update does.

Catching up with the library

A deck imported from the deck library says which release it came from (prov:wasDerivedFrom <…/decks/name/vN.ttl>). When the library publishes a newer release, the deck page (and a course's page) offers to bring the copy up to it (planLibraryUpgrade in domain/libraryUpgrade.ts, shown by LibraryUpgradeContainer.tsx). The Studio lists every copy of an instance, and upgrades several in turn, each the same way (studio.md). A copy of a release added from a link is upgraded the same way too, to the newest version its creator's catalog lists, when it can be read (deck-library.md).

The deck list marks each copy the library has a newer release of with "New release" after its name, its row leading to the deck page and its offer (newerRelease of useCopies in deckTreeEditor.ts). It tells this from the library's index alone, which the list reads anyway when a deck is a copy (libraryCopiesOf, offersNewerRelease in library.ts, as the Studio's library copies screen does): it reads no release and plans no upgrade, which would read every copy's cards and two releases each time the list opens. So it says only what the index tells; a copy the deck page would offer nothing for is marked all the same: one whose newer release changes nothing in it or uses a card format this app does not know, and one of a release the index does not list, which may be no older than the newest. A copy of a release added from a link is not marked: only its creator's catalog would list a newer one.

The plan compares three sets of cards by fragment id — the release the copy came from, the current release, and the copy — so the library's changes reach only what the user left as the library had it:

In the releasesIn the copyThe upgrade
Added in the new releaseNot thereAdds it (retired, if the release has it retired)
ChangedAs the old release had itChanges it
Added, changed or removedChanged by the userKeeps the user's card, and says so
RetiredAnythingRetires it: kept, with its review states, but no longer studied
Retired before, in use againRetiredBrings it back, with the review states it had
RemovedAs the old release had itRemoves it, and its review states
AnythingRemoved by the userLeaves it removed
RemovedNot there any moreNothing to write but dropping the review states left of it; the deck still moves to the release
Added, changed, retired or in use againAlready as the new release has it: content and retirement, or, changed by the user, retirementNothing to write, but the deck still moves to the release

Retiring is not a change of the card's content, so it applies to cards the user changed too: nothing of theirs is lost. A library release never removes a card any more (the build refuses one that drops a card of the release before it); the removal rows are for releases made before cards could be retired.

A copy more than one release behind is compared with the releases in between too: a card as one of them has it is the library's, as though the copy came from that release (an upgrade to it, cut off before it moved the deck's entry, wrote it), so the newer release changes it, and its retirement follows the newer release's.

A card's distractors are part of its content: a release that changes a wrong option, its note or their order changes the card. A course's deck (either release is a course) holds only the cards the learner has answered, so its upgrade adds none; the learner reaches them through the course. It changes, retires and restores the cards it holds as above. A card the release removed that it lacks is no offer there: it lacks every card the learner has not reached. The course's outline (its chapters, steps and theory) is never copied, but read from the release the deck names, so a release that changes only the outline is an offer of its own (LibraryUpgradePlan.outline, compared by sameOutline in domain/course.ts): the upgrade writes no card, and moving the deck to the release brings it. So is a release that changes only questions the learner has not reached (LibraryUpgradePlan.unreached): their content, wrong options or retirement, which the deck meets as the release it names has them. The course's page offers the upgrade too, as the deck's page does.

A card's text format is part of its content too: a release that only marks a card as Markdown changes it. No text format and sm:plainText compare the same, as the vocabulary defines them, so a card switched to Markdown and back is still the library's. One exception tells an older app's doing from the user's: a copy that is as the old release had it but for lacking the release's text format, as an app before vocabulary 1.15 imports or writes it, counts as left alone, both here and when the user settles the deck's languages. The newer release then brings the marker back, whether or not it changes the card otherwise. A copy that states a format other than the release's, sm:plainText against sm:markdown included, was changed by the user and is kept.

A new study direction is taken up when the copy is still studied the old release's way. So are the deck's title, description, keywords and themes, each when the copy still has the old release's; a title or description the user changed is kept, given the languages the release adds only when it says the same as the release in every language both have, sharing at least one (so a copy whose English the user retagged to the language it is really in still agrees in the languages left). Keywords compare language by language, in any order: a copy whose keywords are the old release's in every language takes the new release's, all languages at once (untagged keywords imported from a format-4 release give way to the new release's tagged ones), and a copy whose keywords the user changed in any language keeps them all. The default description the app gave a deck that had none (Flashcards: <title>.) is not the user's: it gives way to the release's description. A copy upgraded before upgrades brought these texts along is given its own release's languages, the same way, once a session when its page opens (addReleaseLanguages): nothing the user wrote changes. The notice lists the notes of every release in between. Nothing is offered for a release that is not newer, uses a card format this app does not know, or would change nothing. A copy with cards already as the newer release has them — content and retirement, or, for a card the user changed, retirement (applied: an upgrade cut off after writing them, or the user's own edits alike) — is still offered the release, which moves its entry and writes nothing to those cards: a deck left naming the older release would have a later upgrade take the newer release's changes for the user's. So is a copy whose direction, title, description, keywords or themes are already the newer release's where the release changed them (appliedAbout), and one that no longer has a card the release removed (gone: removed by an upgrade cut off after writing the cards, or by the user; not a course's deck, as above), whose upgrade drops any review states left of it, with those of the cards it removes. Planned on a copy an upgrade cut off half-way left, the plan thus finishes that upgrade, or takes the copy on to a release out since (How an upgrade is applied). Review history of every card but the removed ones is kept.

How an upgrade is applied

An upgrade writes the deck's documents where they are, each in one write made only if it is still as it was read, in an order that leaves the deck readable after each: its cards, then the review states of the cards it removes, then its catalog entry, which says which release the deck is. A failure stops it where it is, nothing put back; planned again, the upgrade is offered and finishes (domain/deckUpgrade.ts, applyLibraryUpgrade in useCases.ts). The deck keeps its URL, its documents theirs and its cards their ids, so the answer log still names them (statistics count a card by its deck and id, not by its document), and so does any link another app made.

Upgrading a library deck
The upgrade reads the deck, puts its cards, drops the removed cards' review states and moves the entry to the new release, and an error stops it where it is.
The diagram's source
flowchart TD
    accTitle: Upgrading a library deck
    accDescr: The upgrade reads the deck, puts its cards, drops the removed cards' review states and moves the entry to the new release, and an error stops it where it is.
    offer["Notice: what the upgrade changes"] -->|Update to release n| read["1 read: the deck's entry and cards;<br/>the plan again must change the same cards;<br/>the review states, when cards go"]
    read --> cards["2 cards: one PUT of the whole document,<br/>If-Match the read the plan was made from<br/>(none when no card changes)"]
    cards --> reviews["3 reviews: the removed cards' states dropped,<br/>If-Match the read (none when there are none)"]
    reviews --> entry["4 entry: moved to the new release,<br/>with the deck's texts, If-Match the read"]
    entry --> refresh["The page reads everything again at once,<br/>a course's outline and questions included"]
    read & cards & reviews & entry -->|error| stop["Stopped where it is, nothing put back:<br/>the deck readable, still naming the release<br/>it came from; planned again, it finishes"]

On a server that ignores If-Match (node-solid-server), the cards' PUT cannot be held to the read the plan was made from: a card another device edited in between is written over, as by any whole write of a document there.

Proof of the upgrade on a real server

npm run test:pod runs the upgrade against the same servers as the format update, through the app's own use cases and Solid adapters, with every request recorded as it reaches the server (deckUpgrade.integration.test.ts). It seeds an instance holding a copy of a release, studied, its cards shared with a friend, and checks that:

Adding a format version

  1. Write ns/shapes/<class>/v<N+1>.ttl (copy v<N>.ttl, change the @base to its own address and the version assertion to sh:hasValue N+1, add or change the properties). Add any new terms to the vocabulary. A shape's sh:name version must match its file's, with no gaps, so a class whose shape shares a file with another's moves to a folder of its own when only one of them gets a new version: library deck 5 is ns/shapes/library-deck/v5.ttl, a self-contained copy of LibraryDeckV4 from ns/shapes/deck/v4.ttl.
  2. npm run generate: the new record type, descriptor and LATEST_VERSION appear.
  3. Add packages/domain/src/shapes/migrations/<class>/<N>-to-<N+1>.ts and register it in migrations/index.ts.
  4. Adjust <class>FromRecord / <class>ToRecord in packages/domain/src/ if the model changed, and the writers that build the record.
  5. Add fixtures under packages/vocab/fixtures/<class>/v<N+1>/ and the conformance fixture; document the version in the table above.

The chain test, the conformance test, the drift check and the plan and notice tests fail until all of that agrees; nothing else needs touching.