Consolidate Setup Tool, Configuration Tool, and Preloaded Tree Creation

[CarolineDenis]

Fixes #7673
See issue description

(Cloned from #6671)

[emenslin]

1. Setup Flow
Setup Forms

• All setup forms must display correctly in both light mode and dark mode (based on system settings).
• Users can only progress to the next form once all required fields are completed.
• User selections must be reflected in the Overview sidebar in real time.
• The setup flow must be submittable at the end.
• Users must be able to log into the database after setup completes.

Default Data & Configuration

• Default schema configuration captions and descriptions must be populated (not blank).
• Default picklists must be created correctly.
  • Note: The Agent Type picklist does not exist in the schema config, this is expected.
• Default preparation types must be created correctly.
• All tree viewer pages must load without errors.
• The admin user’s agent must be created correctly and must always include a last name.
• An empty Global Preferences file must be created in App Resources.

2. Tree Configuration During Setup
Storage Tree

• Users must be able to modify the storage tree configuration during setup.
• Changes must be correctly reflected in the application after setup.

Geography Tree

• Users must be able to modify the geography tree configuration during setup.
• Both options must work:
  Preloaded tree
  Empty tree

Taxon Tree

• Users must be able to select a preloaded default tree when one exists.
• The selected tree must be correctly preloaded and visible when the app opens.

Geology Discipline

• Relevant tectonic and chronostratigraphic (chrono) trees must be created automatically.

3. Tree Viewer: Preloaded & Empty Trees

• Users can create an empty tree from the tree viewer.
• Users can modify rank configuration in the tree viewer.
• Users can create a preloaded tree from the tree viewer.
• Empty trees now have an upload button. The Upload Tree icon under root creation must work as expected.
• Users can delete a rank with no associated nodes.
• If ranks are deleted and a tree is then loaded, the tree configuration must be respected.
• Users receive notifications for:
  Tree creation start
  Successful completion
  Errors or cancellation

4. Configuration Tool
Institution Tree Viewer

• Users cannot access config tool if they are not an institution admin.
• Users can click and drag to view all nodes in the institution tree.
• Clicking a resource node opens the resource form.
  • Users cannot create a discipline with a discipline name that is already in use.
• Users can:
  • Edit a resource
  • Create new resources
  • Delete collections (created in 7)
• Users cannot create collections through the subview in a discipline form.
• Users can collapse and expand sections in the left panel.
• Users can switch between vertical and horizontal views in the institution tree viewer.

---

Trees

• lithostrat doesn't have any ranks selected as being in full name, not sure if that i necessarily a problem or not but I wanted to mention it just in case

• When I originally uploaded the minerology tree I ran into this issue (first comment on this review #7674 (review)) but it did eventually upload correctly, but only after a long period of time.
• Not sure if this is related to system config or not, but after creating a new collection and not pre-loading taxon I pressed the populate tree button and selected a tree, it once again got stuck on the tree creation dialog, but after refreshing it said "Tree creation in progress" but I didn't get any notification, and after waiting several minutes it doesn't seem to upload at all. https://blankdbsetup1elizabeth220260219-issue-2931-2.test.specifysystems.org/specify/ (in the invert paleo collection)

System config

• Cannot pre-load a geography tree through the system config, it just adds an empty tree. I did do some customization (i.e. making some ranks not enforced) not sure if that is related or not.
• When creating a second division some of the trees appear to be misnamed (i.e "Taxon" is "taxon"). Not sure if it is related to creating a second division or not but the collections under the first division appear correctly.

• Non institution admin can edit the system config

Details

https://github.com/user-attachments/assets/3cca223f-c6ab-4a23-a8ff-05972bbfa7b9

[Iwantexpresso]

PR-7674

1. Setup Flow
Setup Forms

• All setup forms must display correctly in both light mode and dark mode (based on system settings).
• Users can only progress to the next form once all required fields are completed.
• User selections must be reflected in the Overview sidebar in real time.
• The setup flow must be submittable at the end.
• Users must be able to log into the database after setup completes.

Default Data & Configuration

• Default schema configuration captions and descriptions must be populated (not blank).
• Default picklists must be created correctly.
  • Note: The Agent Type picklist does not exist in the schema config, this is expected.
• Default preparation types must be created correctly.
• All tree viewer pages must load without errors.
• The admin user’s agent must be created correctly and must always include a last name.
• An empty Global Preferences file must be created in App Resources.

2. Tree Configuration During Setup
Storage Tree

• Users must be able to modify the storage tree configuration during setup.
• Changes must be correctly reflected in the application after setup.

Geography Tree

• Users must be able to modify the geography tree configuration during setup.
• Both options must work:
  Preloaded tree
  Empty tree

Taxon Tree

• Users must be able to select a preloaded default tree when one exists.
• The selected tree must be correctly preloaded and visible when the app opens.

The taxon tree for minerals is taking an extremely long time to populate itself properly and is displaying this view as its default when first generated 



Tested this in a different discipline and it both lingers on geology specifically for much longer but also does not display this message


Geology Discipline

• Relevant tectonic and chronostratigraphic (chrono) trees must be created automatically.

3. Tree Viewer: Preloaded & Empty Trees

• Users can create an empty tree from the tree viewer.
• Users can modify rank configuration in the tree viewer.
• Users can create a preloaded tree from the tree viewer.
• Empty trees now have an upload button. The Upload Tree icon under root creation must work as expected.
• Users can delete a rank with no associated nodes.
• If ranks are deleted and a tree is then loaded, the tree configuration must be respected.
• Users receive notifications for:
  Tree creation start
  Successful completion
  Errors or cancellation

when using the upload option the tree upload button is seemingly progressing, but whenever I attempt to start other uploads through this button they display no progress and whenever minimized do not send out the notification "The default tree creation process has started."

Details

https://github.com/user-attachments/assets/9800be3c-b3fd-4dab-83b6-efdfad9311f0

 Whenever using the cancel button the operation has been cancelled notifications is properly sent, but the start notification is still missing


Details

https://github.com/user-attachments/assets/4fc2e8b0-c454-4e8f-800a-d2b079aea8fd

4. Configuration Tool
Institution Tree Viewer

• Users cannot access config tool if they are not an institution admin.
• Users can click and drag to view all nodes in the institution tree.
• Clicking a resource node opens the resource form.
  • Users cannot create a discipline with a discipline name that is already in use.
• Users can:
  • Edit a resource
  • Create new resources
  • Delete collections (created in 7)
• Users cannot create collections through the subview in a discipline form.
• Users can collapse and expand sections in the left panel.
• Users can switch between vertical and horizontal views in the institution tree viewer.



if you copy paste the correct path (it should be: https://"your-db-name".test.specifysystems.org/specify/system-configuration/) into the search engine then the user without proper permissions can access and view the system config tool. Thankfully They are still permission blocked rom modifying and interacting with any of the records present here. I marked this as a fail for now since they have viewing access. This should be a security/privacy concern if an unintended user can observe the institutional hierarchy because that information can be used for malicious purposes, making this a mild vulnerability.





[CarolineDenis]

Summary of last reviews

• lithostrat doesn't have any ranks selected as being in full name (Acceptable) (@emenslin )
• Mineralogy tree takes a long time to load (Will not be fixed in this release) (@emenslin + @Iwantexpresso )
• Preloading tree gets sometimes stuck but end up loading (Will not be fixed in this release) (@emenslin )
• when using the upload option the tree upload button is seemingly progressing, but whenever I attempt to start other uploads through [..] (@Iwantexpresso ) Will not be fixed in this release
  • This is a limitation with the background worker, the upload will start as soon as the previous one finishes.
• Whenever using the cancel button for upload tree, the operation has been cancelled notifications is properly sent, but the start notification is still missing
 (Expected) (@Iwantexpresso )

TODO

• If you customize geography tree in system config during discipline creation, an empty tree is being created even when asked to preload (@emenslin)
• When creating a second division some of the trees appear to be misnamed (i.e "Taxon" is "taxon"). (@emenslin)
  • The schema config is being created in the background, so it should appear correctly after refreshing once all background tasks are complete
• Non institution admin can edit the system config (@emenslin), currently the code allows people that have read write etc privilege for Institutions, collections etc table to access it. To change.
• if you copy paste the correct path (it should be: https://"your-db-name".test.specifysystems.org/specify/system-configuration/) into the search engine then the user without proper permissions can access and view the system config tool (@Iwantexpresso )

[emenslin]

1. Setup Flow
Setup Forms

• All setup forms must display correctly in both light mode and dark mode (based on system settings).
• Users can only progress to the next form once all required fields are completed.
• User selections must be reflected in the Overview sidebar in real time.
• The setup flow must be submittable at the end.
• Users must be able to log into the database after setup completes.

Default Data & Configuration

• Default schema configuration captions and descriptions must be populated (not blank).
• Default picklists must be created correctly.
  • Note: The Agent Type picklist does not exist in the schema config, this is expected.
• Default preparation types must be created correctly.
• All tree viewer pages must load without errors.
• The admin user’s agent must be created correctly and must always include a last name.
• An empty Global Preferences file must be created in App Resources.

2. Tree Configuration During Setup
Storage Tree

• Users must be able to modify the storage tree configuration during setup.
• Changes must be correctly reflected in the application after setup.

Geography Tree

• Users must be able to modify the geography tree configuration during setup.
• Both options must work:
  Preloaded tree
  Empty tree

Taxon Tree

• Users must be able to select a preloaded default tree when one exists.
• The selected tree must be correctly preloaded and visible when the app opens.

Geology Discipline

• Relevant tectonic and chronostratigraphic (chrono) trees must be created automatically.

3. Tree Viewer: Preloaded & Empty Trees

• Users can create an empty tree from the tree viewer.
• Users can modify rank configuration in the tree viewer.
• Users can create a preloaded tree from the tree viewer.
• Empty trees now have an upload button. The Upload Tree icon under root creation must work as expected.
• Users can delete a rank with no associated nodes.
• If ranks are deleted and a tree is then loaded, the tree configuration must be respected.
• Users receive notifications for:
  Tree creation start
  Successful completion
  Errors or cancellation

4. Configuration Tool
Institution Tree Viewer

• Users cannot access config tool if they are not an institution admin.
• Users can click and drag to view all nodes in the institution tree.
• Clicking a resource node opens the resource form.
  • Users cannot create a discipline with a discipline name that is already in use.
• Users can:
  • Edit a resource
  • Create new resources
  • Delete collections (created in 7)
• Users cannot create collections through the subview in a discipline form.
• Users can collapse and expand sections in the left panel.
• Users can switch between vertical and horizontal views in the institution tree viewer.

---

Trees

• Only the Botany (Tracheophyta) has Division, the other botany trees have phylum, I'm not sure which one it should be but they should probably all be the same

Details

02-23_09.38.mp4

• I also briefly ran into an issue where I uploaded the Botany (Tracheophyta) tree on setup and then I uploaded the other 2 botany trees in the same discipline and when I reopened the Botany (Tracheophyta) it said "tree creation in progress." I wasn't able to recreate it but I did get this screenshot where you can see there are other trees created afterwards. Link to DB: https://blankdbsetup1elizabeth20260223-issue-2931-2.test.specifysystems.org/specify/

• Although I don't think we need as many notifications as we had before, I do think some notifications are needed so you can tell if a tree is being uploaded upon setup, some of the trees take awhile, especially when preloading geography and taxon and it can be hard to tell what's going on without any notifications. Maybe just one should happen when the trees finish?
• Sometimes when trying to upload a prepopulated tree I don't receive any notifications and it gets stuck on the "default tree creation started" dialog. Sometimes after refresh the notification does appear but sometimes I don't get it at all.

• Not sure if it's related or not but I also ran into this (similar to what happens with the minerology tree on initial setup) with the Fungi tree, populating it from the empty tree viewer after not receiving notifications at first.

Details

02-23_11.09.mp4

System Config

• After creating a new herpetology discipline the taxon tree was uploaded before the geography tree

Details

02-23_09.50.mp4

- Chronostrat tree was not populated after creating a new discipline. It does seem to be created with initial setup though

• Not an issue but we should consider letting non-institution admin view the system config with read-only permissions in the future

[Iwantexpresso]

PR-7674-review 2

1. Setup Flow
Setup Forms

• All setup forms must display correctly in both light mode and dark mode (based on system settings).
• Users can only progress to the next form once all required fields are completed.
• User selections must be reflected in the Overview sidebar in real time.
• The setup flow must be submittable at the end.
• Users must be able to log into the database after setup completes.

Default Data & Configuration

• Default schema configuration captions and descriptions must be populated (not blank).
• Default picklists must be created correctly.
  • Note: The Agent Type picklist does not exist in the schema config, this is expected.
• Default preparation types must be created correctly.
• All tree viewer pages must load without errors.
• The admin user’s agent must be created correctly and must always include a last name.
• An empty Global Preferences file must be created in App Resources.

Known Issue (Out of Scope):
The Accounts page and Stats page currently throw errors. These will be addressed separately in
Check default database constraint differences between Sp6 and Sp7 (#7551).

2. Tree Configuration During Setup
Storage Tree

• Users must be able to modify the storage tree configuration during setup.
• Changes must be correctly reflected in the application after setup.

Geography Tree

• Users must be able to modify the geography tree configuration during setup.
• Both options must work:
  Preloaded tree
  Empty tree

Taxon Tree

• Users must be able to select a preloaded default tree when one exists.
• The selected tree must be correctly preloaded and visible when the app opens.

Geology Discipline

• Relevant tectonic and chronostratigraphic (chrono) trees must be created automatically.

3. Tree Viewer: Preloaded & Empty Trees

• Users can create an empty tree from the tree viewer.
• Users can modify rank configuration in the tree viewer.
• Users can create a preloaded tree from the tree viewer.
• Empty trees now have an upload button. The Upload Tree icon under root creation must work as expected.
• Users can delete a rank with no associated nodes.
• If ranks are deleted and a tree is then loaded, the tree configuration must be respected.
• Users receive notifications for:
  Tree creation start
  Successful completion
  Errors or cancellation
  Note: New error found: when the upload button is used the page does not reload, Because of this you are allowed to click the create empty tree button. Resulting in the following tree setup below. It is not a critical issue and functionality wise nothing is broken, but this is still worth mentioning:


4. Configuration Tool
Institution Tree Viewer

• Users cannot access config tool if they are not an institution admin.
• Users can click and drag to view all nodes in the institution tree.
• Clicking a resource node opens the resource form.
  • Users cannot create a discipline with a discipline name that is already in use.
• Users can:
  • Edit a resource
  • Create new resources
  • Delete collections (created in 7)
• Users cannot create collections through the subview in a discipline form.
• Users can collapse and expand sections in the left panel.
• Users can switch between vertical and horizontal views in the institution tree viewer.

I have found no further issues besides the previously named exceptions, and one minor bug I've found in relation to tree upload and create empty tree button. looks good so far.

[bhumikaguptaa]

1. Setup Flow
Setup Forms

• All setup forms must display correctly in both light mode and dark mode (based on system settings).
• Users can only progress to the next form once all required fields are completed.
• User selections must be reflected in the Overview sidebar in real time.
• The setup flow must be submittable at the end.
• Users must be able to log into the database after setup completes.

Default Data & Configuration

• Default schema configuration captions and descriptions must be populated (not blank).
• Default picklists must be created correctly.
  • Note: The Agent Type picklist does not exist in the schema config, this is expected.
• Default preparation types must be created correctly.
• All tree viewer pages must load without errors.
• The admin user’s agent must be created correctly and must always include a last name.
• An empty Global Preferences file must be created in App Resources.

2. Tree Configuration During Setup
Storage Tree

• Users must be able to modify the storage tree configuration during setup.
• Changes must be correctly reflected in the application after setup.

Geography Tree

• Users must be able to modify the geography tree configuration during setup.
• Both options must work:
  Preloaded tree
  Empty tree

Taxon Tree

• Users must be able to select a preloaded default tree when one exists.
• The selected tree must be correctly preloaded and visible when the app opens.

Geology Discipline

• Relevant tectonic and chronostratigraphic (chrono) trees must be created automatically.

3. Tree Viewer: Preloaded & Empty Trees

• Users can create an empty tree from the tree viewer.
• Users can modify rank configuration in the tree viewer.
• Users can create a preloaded tree from the tree viewer.
  I uploaded trees a few times(ichthyology and invertebrate zoology), and every single time I received the notifications of the tree-building process starting and ending. Still, the tree view remained like the image below:

• Empty trees now have an upload button. The Upload Tree icon under root creation must work as expected.
• Users can delete a rank with no associated nodes.
• If ranks are deleted and a tree is then loaded, the tree configuration must be respected.
• Users receive notifications for:
  Tree creation starts
  Successful completion
  Errors or cancellation

4. Configuration Tool
Institution Tree Viewer

• Users cannot access the config tool if they are not an institution admin.
• Users can click and drag to view all nodes in the institution tree.
• Clicking a resource node opens the resource form.
• Users cannot create a discipline with a discipline name that is already in use.
• Users can:
  • Edit a resource
  • Create new resources
  • Delete collections (created in 7)
• Users cannot create collections through the subview in a discipline form.
• Users can collapse and expand sections in the left panel.
• Users can switch between vertical and horizontal views in the institution tree viewer.

Link: https://blankdbsetup2b20260223-issue-2931-2.test.specifysystems.org/specify/

Everything works as expected, except the preloaded tree view.

[alesan99]

Fixed so far:

• Fix order of Geography and Taxon tree creation (Config tool now does Geography -> Taxon)
• Changed all botany division ranks to phylum (After a discussion with @grantfitzsimmons )
• Fix Chronostrat tree not being created in new disciplines

Need to test:

• Possibly fixed incorrectly displaying the "Tree is loading, please wait..." message.
• Possibly fixed being able to import into an existing tree multiple times. It should refresh after closing the tree progress dialog, but I haven't fully tested it yet.

TODO:

• The rest of the issues reported in the reviews

[acwhite211]

I pushed a fix to fix the error on the stats page query. I not seeing any errors on the Accounts page.

Changed the query in get_percent_georeferenced from

SELECT CASE WHEN
  (SELECT COUNT(*)
   FROM locality
   JOIN discipline ON locality.DisciplineID = discipline.DisciplineID
   WHERE discipline.DisciplineID = 3) = 0
THEN 0
ELSE ((COUNT(localityid) * 1.0) /
      ((SELECT COUNT(*) FROM locality) * 1.0)) * 100.0
END AS PercentGeoReferencedLocalities
FROM locality
WHERE latitude1 IS NOT NULL;

to

SELECT CASE WHEN totals.total_count = 0 THEN 0
ELSE (totals.georeferenced_count * 100.0) / totals.total_count
END AS PercentGeoReferencedLocalities
FROM (
  SELECT
    COUNT(*) AS total_count,
    SUM(CASE WHEN Latitude1 IS NOT NULL THEN 1 ELSE 0 END) AS georeferenced_count
  FROM locality
  WHERE DisciplineID = 3
) AS totals;

[emenslin]

Summary of existing issues:

• Sometimes missing notifications and stuck on "default tree creation started" dialog when uploading a prepopulated tree.
• Fungi and Minerology tree not loading properly after initial upload
• Chronostrat tree is now populated after creating a new discipline, however, it is very slow and takes several minutes to load, even if not prepopulating geography or taxon trees.
• Possibly fixed incorrectly displaying the "Tree is loading, please wait..." message.
• Possibly fixed being able to import into an existing tree multiple times. It should refresh after closing the tree progress dialog, but I haven't fully tested it yet.
• When creating a new discipline "Collection Object" is misnamed "collectionobject", I imagine this might be related to the misnamed tree issue mentioned here (although in this case it was a different discipline not division), but I think users should not be able to open up a collection if the schema config hasn't been created yet, I feel like this could lead to even more problems.

• We need to consider having some sort of indication that various processes are happening, whether that be notifications or different wording. Right now if you open up a new discipline the schema config might still be in the process of being created (see previous point), however, if you pre-loaded a geography and taxon tree and look at the each of them, they will both say they're being created even though they haven't even started yet. I feel like this could be very confusing to users, especially when certain trees take a really long time to upload which can be very misleading.

[grantfitzsimmons]

Responding to @emenslin's last comment

We need to consider having some sort of indication that various processes are happening, whether that be notifications or different wording. Right now if you open up a new discipline the schema config might still be in the process of being created (see previous point), however, if you pre-loaded a geography and taxon tree and look at the each of them, they will both say they're being created even though they haven't even started yet. I feel like this could be very confusing to users, especially when certain trees take a really long time to upload which can be very misleading.

I wonder why users can start using the database before all resources are created. Is there a benefit to this now that tree creation is much faster? It seems they will encounter unfinished sections of the collection/discipline, such as incomplete trees and/or unfinished schema config generation. The advantage of using the app early seems to be outweighed by the downsides.

[Iwantexpresso]

HI Here is a comment expanding on a smaller issue I have been able to find, This is related to the populate tree with default records button and initialize empty tree button.

when the populate button is first clicked on, you are allowed to click the create empty tree button as well.
 Here is a video better showing the actions that can be taken:
https://github.com/user-attachments/assets/1d8e884e-3589-4611-aa1c-51f42d4d7562
Resulting in the following tree setup below having both a life and a root node at its origin.

As I have briefly mentioned before I don't believe it to be a critical issue and functionality wise nothing is broken, but it is still worth mentioning to keep a record of its existence. I have briefly discussed this with Alejandro and a possible solution could be to make the page reload after clicking the close button on the default tree creation, the one shown below, since this would properly set the please wait until default tree is created option.


[CarolineDenis]

Decision to fix most of mentioned issues:

• At first run (Guided Setup), we should not let the user work in the app until all necessary resources are created.

• When using System Config and adding a discipline, we don't make it possible to log into the new collection until all resources are ready.
  It shouldn't even appear in the collection list until it is done.

Once this is implemented, we can remove the guard in the tree viewer with the warning letting the user know that a tree is being created.
All codes on frontend and backend that were added for this feature can also be removed.

[acwhite211]

Sounds good. I think the simplest way to determine that it is all done with the setup process is to check when the worker task is completed.