IE-0001: Directory format for extensions with resources by curiousdannii · Pull Request #1 · ganelson/inform-evolution (original) (raw)

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service andprivacy statement. We’ll occasionally send you account related emails.

Already on GitHub?Sign in to your account

Conversation16 Commits1 Checks0 Files changed

Conversation

This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.Learn more about bidirectional Unicode characters

[ Show hidden characters]({{ revealButtonHref }})

Proposal: IE-0001
Authors: Graham Nelson
Language feature name: None
Status: Draft
Related proposals: None
Implementation: None

Summary

Substantially expanding the range of resources an extension can provide.

curiousdannii changed the title~~IE-0001 Directory format for extensions with resources~~ IE-0001: Directory format for extensions with resources

Aug 1, 2022

Multi-file extensions are a good idea and will surely help with quite a few things. Here are some of my thoughts about them:

Being able to include a kit in an extension is great. We don't want authors to have to manually edit external text files in order to load kits.
Putting the version number in the extension folder name: are you envisioning including multiple versions of an extension in one project, or is this just to make it easier when one project wants version 1 and another wants version 2?
Node and Rust both let you load multiple versions of a package in one program, and it often works because their packages are frequently fairly isolated functions that you call manually. But they can trip you up: a class from one version often won't be identified as the same as a class from another version, and if an extension keeps some global state for itself, then loading two versions means you'll get two sets of the global state. I7 would be even worse: loading two versions of an extension would duplicate all the rules they define, and which version's phrase would be used would be unpredictable. Using multiple versions of an extension in one project should not be a goal IMO.
Personally, I don't think indoc is a great format for writing and transferring documentation. I'd prefer docs be written in Markdown instead (the Github dialect so that tables are an option.) But maybe it would be possible to support both, distinguished by file extension?
What happens if an extension includes a copy of Parchment? Or what happens if two extensions include a new Template but use the same name? Which one will be used?

I'm not sure what is included in the metadata currently, but useful things to include could be:

Compatible Inform releases
Author contact details
Website
License
Where to report bugs
Possibly a field filled in by Inform adding where it was downloaded from
How to test (test command, extension, project, etc)
Dependencies?

Metadata could also communicate things like whether an extension needs to be output to Glulx or whether it can be converted to C code etc. The user will discover problems when they try to compile it, but if it was in the metadata, then maybe the incompatibility could be raised earlier, and maybe the error message shown could be less obscure?

Ideally, even "directories" of multiple files would still be a single file to install from the user's perspective -- e.g. to install an extension they just download a single .zip file and put it in their extensions folder. This also helps to keep the component files locked together and reduce version incompatibility weirdness.

Within an Extension Project the files should be loose for easy editing and the install/publish step could zip them up.

Having said that, within git repositories (such as the current I7 Extensions repo) it's nicer to have loose files as well, so that you can more readily see text diffs of the changes -- I haven't tested it, but I doubt GitHub would look into the contents of the zip to diff that nicely, and it would certainly inflate repository sizes faster than it should. So there's still a bit of logistic challenge there, unless the process of downloading a single extension from GH (and storing it as a zip) is automated in the IDEs, or both formats are supported so that the current practice of using the whole GH repo as your Extensions folder still works.

i'm looping back here to better understand IE-0011 and IE-0003. since i don't fully understand the .inform/.materials distinction, i have a few clarifying questions:

would a separate IE for distributing extensions outside the main distro be appropriate?
do inter-compiled kits imply the desire to allow distribution of "binary-and-no-source" extensions? (i.e. commercial dev tools)
in use case 7.1, would Boston Fire Dept be able to include annual updates to previously released titles projects as DLC patches to the blorb? the annual dalmatian calendar is for charity, and should not be expensive to re-reproduce, after all.

A somewhat obvious addition here would be the option to have a Tests directory where one would expect a "Red Fire Hydrants.intest" file (I suppose I'm assuming here a mod to intest to be willing and able to ignore the -v.i7xd part of the directory name).

Though I would prefer to not have the version number be part of the directory name. There's an explicit version number in the metadata and I don't see an advantage to redundantly putting it in the directory name. To encourage compliance with semantic version-ing, I think it's significant to ensure that incrementing a version number is painless. Accumulating a collection of old directories provides enough nuisance value that I fear it would discourage version number changes and it fits badly with how a developer would want to use source control. (Perhaps, though, a part of the IDEs' handling of these could include displaying the current version number prominently in a way that would provide the equivalent benefit to the author that the version number in the directory name would have had.)

There are a couple of existing extension examples that don't compile because they need files for a Figure or a Sound.

I briefly considered suggesting that any files needed by the examples would be under the extension's own Materials directory, but I don't like the idea of cluttering that with things needed only by examples. So maybe it would be appropriate for the Documentation directory to also (potentially) have Figures and Sounds subdirectories. (I figure it's acceptable that all of these would be provided to all the examples -- it should be easy to avoid creating conflicts.)

Yes, I think I agree with that. We were going to need Documentation/Figures anyway so that documentation can potentially include diagrams, etc.

Though I would prefer to not have the version number be part of the directory name.

The obvious problem that we need different names to support having multiple versions in a given project finally occurred to me. So, a mini-proposal:

Under the project.materials directory, there may be a Development directory, under which there may be an Extensions directory, under which there may be, say, "Zed Lopez/Info Utility", which is an extension-in-directory-form. Building the project starts with looking at what's in Development and writing the corresponding thing to project.materials/Extensions with the appropriate version number. So, best of all worlds: a developer can keep something under source control nice and neatly and we can have as many distinct versions in a project as we want.

(This scheme could eventually be extended to Templates, if desired.)

So, I'm afraid I do very much want it to be the case that the canonical filename for a single-file extension should include the version number, and similarly for the canonical directory name. It's not just for the materials folder, where I agree, it may be a slight nuisance for version-control. It's also for all of the other potential places where inbuild may be handling things, such as archives of extensions. The only reliable way to store extensions is for the namespace of potential destinations to be able to represent faithfully the full range of potential extension versions.

But with that said, Inform does read extensions which don't have the version number in the filename. If you drag a directory extension into your materials folder by hand, it can have the wrong version number or none, and Inform will detect and correct this. You can put more or less anything into the installer, and it will sort out what the naming should be and will apply it. So the zip files being served by the Public Library, for example, can be named however the PL likes, and that won't affect the ability of the installer to work it all out.

And besides that: (i) It's not like people change version numbers for extensions every day, and (ii) Version control deals with file renames all the time. If you do change a version number, or change the version of an extension you're using, that's absolutely a change which version control should be registering.

I may be over-making my case here, but it's just such a potential nest of vipers to start making exceptions. For example, the obvious exception would be to not bother with this version-numbers-in-the-title business for extensions in the materials folder (or, as Zed suggests, some special enclave of it). That implicitly imposes that a project can't have two versions installed - well, we could live with that. Except of course it doesn't, unless we actually forbid people putting number-in-name extensions there. But then people can't drag otherwise valid extensions into their own projects any more. So... we don't want to forbid that. But then, we end up with a situation in which extensions are being developed under filenames which are vague as to their content. When those extensions are distributed, maybe those filenames are still vague, because users, having no special incentive, haven't amended those names. So when the extensions are used in some other person's materials folder - supposing they are moved directly, not via the installer - now names can collide, and unpredictable things will happen. Inform's current practice of auto-maintaining the namespace of the extensions in the materials folder so that it is always correct seems to me the only safe way through what really is something of a swamp,

I may not have communicated clearly: my proposal has none of the drawbacks you describe. Development/Extensions would not be an auxiliary extensions directory and its contents would never directly be built into a project. By my proposal, a new first step in compilation would be to check the Development/Extensions directory and write the corresponding extension into the real Extensions directory with the appropriate versioned directory name. This could be implemented by plainly copying Development/Extensions' contents to Extensions. The existing implementation's correction of any version-less directory names would then rewrite them to include the version.

I'm not suggesting a change to the auto-maintenance of the extension namespace in the Extensions dir proper.

My proposal creates no risk beyond whatever slight risk is inherent in the existing proposal. Even if someone drags and drops a misnamed extension into the Development/Extensions directory, the end-result is the same as if they'd dropped it in their Extensions directory: they end up with something whose name has been corrected.

All of the tools can continue to consider misnamed extensions as directories-non-grata without change, with the sole exception being Development/Extensions. The (small) population of extension authors can be told that what they share should have the versioned name. It's straightforward to enforce by policy that things without correctly versioned names won't be accepted into the Friends repo or the Public Library.

Will inbuild be able to build .i7x files from .i7xd directories (necessarily omitting Figures, Images, Sounds), or to deconstruct .i7x files to .i7xd ? (or to turn either into Public Library-friendly zip files?)

Hmm. Okay, I suppose git doesn't care about the name of a repo's directory and it wouldn't interfere with source control or matter one way or another for that name to keep changing due to corrections by inbuild.

I still like my suggestion for a Development dir, but I withdraw my making this out to be a big deal. I'll use any spare big-deal-making vis-a-vis source control-friendliness this leaves me to agitate for allowing the .materials directory to be the only thing in need of tracking. 😀

And besides that: (i) It's not like people change version numbers for extensions every day, and (ii) Version control deals with file renames all the time. If you do change a version number, or change the version of an extension you're using, that's absolutely a change which version control should be registering.

Technically, any edit that you make should include a bump to the version number (though in practice it probably doesn't matter unless you release the change to others -- but that line gets blurry if your release is just a git push). Also, git in particular only has fuzzy tracking of file renames -- if you simultaneously rename the file and make significant enough changes to its content (and this could even just be changing the indentation or rearranging some of the section ordering) then it will treat it as a unique file and you'll lose the history tracking without some heroics.

When those extensions are distributed, maybe those filenames are still vague, because users, having no special incentive, haven't amended those names. So when the extensions are used in some other person's materials folder - supposing they are moved directly, not via the installer - now names can collide, and unpredictable things will happen.

Isn't the solution to that to define a predictable rule for that case? Hopefully, people would use Extension Projects, and publishing from the EP to the main Extensions folder could automatically append/update the version -- which is good, as it allows multiple versions to exist side-by-side in the Extensions folder and in repositories like the PL or Friends github -- but less good in that adding new files isn't even a rename so they'll have disconnected history. (You can still diff them explicitly but you won't be able to do a blame. But if it's a binary format like a zip then that may not matter anyway.)

So, what other cases do we have? If there's a private version in the Materials folder then that should pretty much always win over any version in the main Extensions folder (or produce a Problem if it doesn't match the explicitly included version in the story). There can't be two different versions in the Materials folder unless they do have the version included explicitly in the filename; then you just need to have a rule that if the story specifies a version explicitly then that's what it gets, otherwise it gets the highest one, or something like that. (Which is the same rule you'd use for two versions in the main Extensions with nothing private.) No unpredictability.

I'm assuming that when you build your internal list of extensions, you use the versioned name as your lookup key regardless of whether the file was named that way or not, allowing you to keep side-by-side refs. This might require opening the file and parsing the first few lines, but that should still be reasonably fast, and can be made faster by caching it along with metadata (e.g. modification date) so that you know you don't have to re-parse it.

Graham wrote, incorrectly:

And besides that: (i) It's not like people change version numbers for extensions every day, and (ii) Version control deals with file renames all the time.

This is not correct. I just want to confirm what uecasm said. First, you should be changing the version number EVERY TIME. Extension version numbers should, in fact, be changing constantly.

While major version numbers might not change so often, since they reflect an interface, patchlevel should be changing on every upload. And, Git only does fuzzy pattern matching and will not always ID the file as the same file if you're changing its name simultaneously with changing its contents substantially.

I am writing a Java-based IDE for Inform7, and I thought to come to the Inform-Evolution repository to make a comment here. It would be nice for the command-line Inform tooling to provide an API or tools for creating and validating Inform7 project directory layouts, such as "asdf.inform" and "asdf.materials".

It is not necessary, but it would be a convenience that may ease development of other tooling Inform users have created, such as other CLI.

I cannot say I have thoroughly prepared myself for making an IDE for Inform, but I have taken a glancing look at the GNOME IDE and with dissatisfaction from its installation process, especially on older machines, I thought a Java IDE would be a better solution (given that Java "should be" more portable).

I7 can validly run with no materials directory at all; if it's missing when the compiler begins, then it is created. In any case, an empty directory called Whatever.materials is valid, because all contents and subdirectories are optional.

I cannot say I have thoroughly prepared myself for making an IDE for Inform, but I have taken a glancing look at the GNOME IDE and with dissatisfaction from its installation process, especially on older machines, I thought a Java IDE would be a better solution (given that Java "should be" more portable).

I would be happy to try to fix the installation process (depending on how old of machines we are talking about...) Please let me know what the problem was at https://inform7.atlassian.net/jira/software/c/projects/I7/issues.

Labels

formal-proposal

A formal proposal that has been accepted for consideration by the core Inform team