Adam's Lair Forum

game development and casual madness
It is currently 2019/04/24, 18:43

All times are UTC + 1 hour [ DST ]




Post new topic Reply to topic  [ 18 posts ]  Go to page 1, 2  Next
Author Message
 Post subject: [RELEASED] Data plugin
PostPosted: 2014/10/30, 16:51 
Forum Addict
Forum Addict
User avatar

Joined: 2013/09/19, 14:31
Posts: 883
Location: Italy
Role: Hobbyist
Good news everyone! I have almost completed the first version of my new and amazing Data plugin

Where to get it:
  • Source code: https://github.com/SirePi/duality-data
  • Directly inside the Duality Editor's Package management as SnowyPeak's Data Plugin (Core) and SnowyPeak's Data Plugin (Editor) (just add the editor, the core will be downloaded as a dependency)

What it is:
it's a plugin which allows you to load text-based files as Resources, to be used as desired (translations, dialogs, ...) inside Duality.

What it does:
for the moment it lets you import and create txt, xml and xsd files, and csv files; automatically detects changes in the files and reloads them when necessary. The Resource's content are available in the aptly-named Content property as a string, depending on the type of the source file, via one ore more properties and/or methods.

XmlData Resources are validated xml/xsd pairs whose contents are available from the XmlDocument property as an XmlDocument. Validation is performed whenever the xml or xsd are changed and eventual errors are displayed in the editor's log panel.

CsvData Resources are "validated" (a check is performed to verify that all the rows of the file contain the same number of fields) and their contents are available as a matrix that you can access in a number of combinations via Row, Column, Index (the first column of each row, if unique, can be used as a searchable index), Field (the contents of the first row, if unique, can be used as field names for the other rows), or any combination of Row/Index and Column/Field.

JsonData Resources are validated json files whose contents are available from the JsonObject property as a dynamic object. All is managed internally in .net4 and no external dependencies are necessary.

All files can be opened directly in their default editor Notepad using the new Open File contextual menu command by doubleclicking and, when saved, the linked Resources are automagically reloaded and eventually revalidated.


and here it is, in all its glory:
Image

When will it be ready?
Soon... as soon as I understand how NuGet works (in case I can't, expect to see the dll pop up somewhere along with the source code)
IT IS!

What next?
As always I am open to suggestions.. binary data import? csv parsing?

Can it be extended?
Of course! All classes that are public are open for grabs. You can use them as a building block to develop your own custom file importer. Just extend the base class depending on the type of file you want to use as source (txt, xml or csv) and override method AfterReload (it is called everytime the file is loaded - be it first creation or after a change in the contents) and parse the file as you prefer.
After that, just write your own FileImporter to be able to manage your custom file name and that's it :)

@Adam (better? :mrgreen: )
Let's say I want to be able to configure my plugin by saving the default editor's path (i.e. Notepad++ instead of Notepad) somewhere; what would be the best choice? something in the CustomData property of DualityAppData? or another type of AppData? or something else?

[EDIT] Sorry, I was editing my question while you were posting your answer..


--- UPDATES ---

[08/11/2014] First release. Thanks to everyone (especially Adam, BraveSirAndrew and existable) for their help in putting together all the pieces of the puzzle. Hope this will be useful to someone.

[09/11/2014] Fixed the issues described by Adam. Still no icon though :D

[29/01/2015] Added support for json files. And still no icon :redface:

[01/02/2015] Compliant with DualityEditor 1.3.0

[04/02/2015] Added a couple extra utility methods for JsonDynamicObjects.

[03/11/2015] Upgraded for Duality 2.x; Validation of xml via xsd is not available anymore, due to the inability to have xml/xsd validation in PCLs.

_________________
Come on Duality's Discord channel. We have cookies! :mrgreen:


Last edited by SirePi on 2015/02/04, 17:08, edited 13 times in total.

Top
 Profile  
 
 Post subject: Re: [WIP] Data plugin
PostPosted: 2014/10/30, 17:44 
Site Admin
Site Admin
User avatar

Joined: 2013/05/11, 22:30
Posts: 2071
Location: Germany
Role: Professional
SirePi wrote:
@Adam (better? :mrgreen: )
Let's say I want to be able to configure the default editor (i.e. Notepad++ instead of Notepad); what would be the best choice? something in the CustomData property of DualityAppData? or another type of AppData? or something else?


As of now, Duality simply uses Windows defaults to open all external files. This will definitely change at some point in the future via a centralized editor options menu, but there is no ETA on that and this is also not a priority right now. Until then, feel free to roll your own solution when you consider it necessary - just don't grow too accustomed to it as there may be an official API in the future to move to.

However, DualityAppData is definitely not a good place to put something like this. This is reserved for actual game data, not editor data. The editor provides its own set of configuration methods, probably the most fitting one is EditorUserData.xml. You can store your own stuff in there by overriding SaveUserData and LoadUserData in your EditorPlugin class. Most default ones do it, just pick one as an example :)


SirePi wrote:
All files can be opened directly in Notepad using the new Open File contextual menu command and, when saved, the linked Resources are automagically reloaded and eventually revalidated.


Awesome :) You don't need a context menu entry though, a simple double-click would be more intuitive I think. You can do it using something like this:

Code:
public class EditorActionOpenXmlFile : EditorSingleAction<Resource>
{
   public override string Description
   {
      // Some nice display text like "Opens the fancy XML stuff externally"
      get { return YourPluginRes.ActionDesc_OpenXmlFileExternal; }
   }

   public override bool MatchesContext(string context)
   {
      // This is meant for "Open Resource" actions, e.g. doubleclicking them
      return context == DualityEditorApp.ActionContextOpenRes;
   }
   public override bool CanPerformOn(Resource obj)
   {
      if (!base.CanPerformOn(obj)) return false;
      if (obj is XmlFile) return true;
      return false;
   }
   public override void Perform(Resource obj)
   {
      XmlFile xmlFile = obj as XmlFile;
      if (xmlFile != null) FileImportProvider.OpenSourceFile(xmlFile, ".xml", xmlFile.SaveXmlData);
   }
}


You can find the default implementation for Pixmaps, Shaders, etc. here.

SirePi wrote:
The Resource's content are available in the aptly-named Content property as a string.


I would suggest to use the invisible flag so the content isn't even visible in the Object Inspector directly. Just imagine someone pulling a 10000 Kb XML monster in there. Which is an actual case to consider for huge data sets! You really wouldn't want the editor to get stuck putting that into the Object Inspector.

SirePi wrote:
In addition, you can create XmlData Resources, which are validated xml/xsd pairs whose contents are available from the XmlDocument property as an XmlDocument. Validation is performed whenever the xml or xsd are changed and eventual errors are displayed in the editor's log panel.


The auto validation is a great addition! If there's manual editing, there will be human errors.

On a different note, I don't get one thing in your general design: Why do you even separate XmlFile and XmlData? I can't really imagine a useful case where you would have an XmlFile without an XmlData attached to it - after all, you will parse that XML anyway at some point - so you might as well merge the two classes and keep everyone's project directory a little more tidy. You could still (optionally) reference an XmlSchema from XmlData, do your validation, allow editing them externally on double-click, etc.

I guess what I'm trying to say is, I don't really see how XmlFile adds any value besides being a source for XmlData and it adds complexity. Just imagine I would've defined PngImageFile and JpegImageFile Resources (raw files), which are referred to by a Pixmap (parsed image)... the project view would be a mess for no added value!

Duality Resources aren't really designed to be "file wrappers" without adding anything. They represent imported data where "import" means the potentially non-trivial step of converting an arbitrary data format to a Duality-specific one. XML isn't really Duality-specific, but XmlData is still exactly what a Resource was made for, while XmlFile doesn't really map to known Duality concepts:

Illustrating this with an example, let's assume you have an XmlData Resource in your project when some outsourced guy comes along and says that he switched tools and can no longer provide XML, but only JSON from now on. No problem! You can write a custom FileImporter that takes a JSON file and converts it to an XmlData Resource, so you can now import both JSON and XML and still get a mostly uniform XmlData, which can validate against your existing XmlSchema. Actually, I have to admit that this specific example makes me cringe a little bit, but I think you get the point :D Resources shouldn't care which kind of file or file format they came from, and raw files should be sources for FileImporters, not Resources on their own.

Whew, that turned out rather lengthy. Sorry for the wall of text, I just wanted to make sure to explain this thoroughly :D And thanks for putting the effort into making this a distinct plugin! It definitely has potential to be a useful tool in the box :)


EDIT: Re-reading this, it feels like I'm bashing design choices in your newly created plugin. This really isn't how this should come along! I just want to point out this thing with the conceptual consistency between your plugin and Duality that I think might have slipped attention otherwise.

_________________
Blog | GitHub | Twitter (@Adams_Lair)


Top
 Profile  
 
 Post subject: Re: [WIP] Data plugin
PostPosted: 2014/10/30, 21:51 
Forum Addict
Forum Addict
User avatar

Joined: 2013/09/19, 14:31
Posts: 883
Location: Italy
Role: Hobbyist
Adam wrote:
As of now, Duality simply uses Windows defaults to open all external files. This will definitely change at some point in the future via a centralized editor options menu, but there is no ETA on that and this is also not a priority right now. Until then, feel free to roll your own solution when you consider it necessary - just don't grow too accustomed to it as there may be an official API in the future to move to.

However, DualityAppData is definitely not a good place to put something like this. This is reserved for actual game data, not editor data. The editor provides its own set of configuration methods, probably the most fitting one is EditorUserData.xml. You can store your own stuff in there by overriding SaveUserData and LoadUserData in your EditorPlugin class. Most default ones do it, just pick one as an example :)[/justify]

I'll think about it.. it's not so important for now

Adam wrote:
Awesome :) You don't need a context menu entry though, a simple double-click would be more intuitive I think. You can do it using something like this:

You mean I could doubleclick on a Resource and its default editor would open? :omg: I never knew that!

Adam wrote:
You can find the default implementation for Pixmaps, Shaders, etc. here.

Copied, but I am using this (all file types - txt, xml, xsd derive from TextFile)

Code:
public override void Perform(TextFile file)
{
  System.Diagnostics.Process.Start("notepad", file.SourcePath);
}

Adam wrote:
I would suggest to use the invisible flag so the content isn't even visible in the Object Inspector directly. Just imagine someone pulling a 10000 Kb XML monster in there. Which is an actual case to consider for huge data sets! You really wouldn't want the editor to get stuck putting that into the Object Inspector.

Done :mrgreen:

Adam wrote:
On a different note, I don't get one thing in your general design: Why do you even separate XmlFile and XmlData? I can't really imagine a useful case where you would have an XmlFile without an XmlData attached to it - after all, you will parse that XML anyway at some point - so you might as well merge the two classes and keep everyone's project directory a little more tidy. You could still (optionally) reference an XmlSchema from XmlData, do your validation, allow editing them externally on double-click, etc.

I guess what I'm trying to say is, I don't really see how XmlFile adds any value besides being a source for XmlData and it adds complexity. Just imagine I would've defined PngImageFile and JpegImageFile Resources (raw files), which are referred to by a Pixmap (parsed image)... the project view would be a mess for no added value!

Ah, that's just a leftover from my first tests (I took the Shader with Vertex and Fragment files as an example); now there is only XmlFile.

Adam wrote:
EDIT: Re-reading this, it feels like I'm bashing design choices in your newly created plugin. This really isn't how this should come along! I just want to point out this thing with the conceptual consistency between your plugin and Duality that I think might have slipped attention otherwise.

No worries :mrgreen: it's good to have someone make me think twice on what I'm doing.

Hope the result will be helpful to someone else :)

_________________
Come on Duality's Discord channel. We have cookies! :mrgreen:


Top
 Profile  
 
 Post subject: Re: [WIP] Data plugin
PostPosted: 2014/10/30, 22:15 
Site Admin
Site Admin
User avatar

Joined: 2013/05/11, 22:30
Posts: 2071
Location: Germany
Role: Professional
SirePi wrote:
Ah, that's just a leftover from my first tests (I took the Shader with Vertex and Fragment files as an example); now there is only XmlFile.


Would you mind to think about renaming it to XmlData? I think XmlData and XmlSchema were excellent names.

Because having "File" in the name of a Resource class suggests the wrong things. In Duality terms, I'm trying to establish the term "Source Files" as "stuff before importing" and "Resources" as "stuff after importing". You usually say: "I've imported this file and now I have this resource". Having a Resource being called something with "File" has a quite confusing potential for newbies when having that in mind. ^^

Quote:
Ah, that's just a leftover from my first tests (I took the Shader with Vertex and Fragment files as an example); now there is only XmlFile.


That's actually something I wanted to get rid of multiple times.. problem is though, a FragmentShader, a VertexShader and a ShaderProgram are all individual OpenGL objects, so not combining them actually might save some graphics resources or allow internal optimizations, because the same VertexShader can be reused for different programs. Still, I'm kind of annoyed by having all those classes:

Code:
VertexShader
FragmentShader
ShaderProgram
DrawTechnique
Material


I mean.. seriously? Not sure whether I can improve on this without getting destructive under certain circumstances though... but it's definitely on "the list".

SirePi wrote:
You mean I could doubleclick on a Resource and its default editor would open? :omg: I never knew that!

Surprise feature! xD

_________________
Blog | GitHub | Twitter (@Adams_Lair)


Top
 Profile  
 
 Post subject: Re: [WIP] Data plugin
PostPosted: 2014/10/30, 22:38 
Forum Addict
Forum Addict
User avatar

Joined: 2013/09/19, 14:31
Posts: 883
Location: Italy
Role: Hobbyist
Adam wrote:
Would you mind to think about renaming it to XmlData? I think XmlData and XmlSchema were excellent names.

Because having "File" in the name of a Resource class suggests the wrong things. In Duality terms, I'm trying to establish the term "Source Files" as "stuff before importing" and "Resources" as "stuff after importing". You usually say: "I've imported this file and now I have this resource". Having a Resource being called something with "File" has a quite confusing potential for newbies when having that in mind. ^^


Not a problem, but this leaves me with the issue of having a TextFile (as in, a simple .txt) Resource. Which is nothing more than a string.. following your reasoning I would be leaning towards going back to have it as an abstract Resource used internally by all the other specializations (xml, json, csv..), but on the other hand, you prefer not to have big strings going around the property editors, and it could always be edited in order to expose also a string[] property called Lines.

Maybe I could just rename it StringData?

Adam wrote:
That's actually something I wanted to get rid of multiple times.. problem is though, a FragmentShader, a VertexShader and a ShaderProgram are all individual OpenGL objects, so not combining them actually might save some graphics resources or allow internal optimizations, because the same VertexShader can be reused for different programs. Still, I'm kind of annoyed by having all those classes:

Code:
VertexShader
FragmentShader
ShaderProgram
DrawTechnique
Material


I mean.. seriously? Not sure whether I can improve on this without getting destructive under certain circumstances though... but it's definitely on "the list".

I don't know.. I like the way it is. The smaller the building block, the higher the chance that the same block could be used by someone else to make something out of it... but it's your call :mrgreen:

_________________
Come on Duality's Discord channel. We have cookies! :mrgreen:


Top
 Profile  
 
 Post subject: Re: [WIP] Data plugin
PostPosted: 2014/10/30, 23:07 
Site Admin
Site Admin
User avatar

Joined: 2013/05/11, 22:30
Posts: 2071
Location: Germany
Role: Professional
SirePi wrote:
Not a problem, but this leaves me with the issue of having a TextFile (as in, a simple .txt) Resource. Which is nothing more than a string.. following your reasoning I would be leaning towards going back to have it as an abstract Resource used internally by all the other specializations (xml, json, csv..), but on the other hand, you prefer not to have big strings going around the property editors, and it could always be edited in order to expose also a string[] property called Lines.

Maybe I could just rename it StringData?


How about PlainTextData? :)

If it's really nothing more than a fancy wrapper for a string, I would consider not to have a direct relationship between this and the other data Resources. (Especially with regards to inheritance hierarchies, which are easily established but hard to maintain and often not the ideal choice.)

The downside of no relation between these classes is potential code duplication, but if that code is really just a single field and property definition, then that's a pretty small downside. On the other hand, the upside would be the ability to specialize each data Resource more towards its actual use: Borrowing your example, PlainTextData could have (editor-invisible) Content and Lines properties without accidentally adding them to XmlData and XmlSchema as well, where they don't really make sense and thus shouldn't be available.

SirePi wrote:
I don't know.. I like the way it is. The smaller the building block, the higher the chance that the same block could be used by someone else to make something out of it... but it's your call :mrgreen:


It's not something that is definitely wrong, or I believe to be bad design, it's just.. bugging me somehow. Maybe there really is a better way to do this, or maybe you're right that it's already fine the way it is.

_________________
Blog | GitHub | Twitter (@Adams_Lair)


Top
 Profile  
 
 Post subject: Re: [WIP] Data plugin
PostPosted: 2014/11/01, 16:25 
Forum Addict
Forum Addict
User avatar

Joined: 2013/09/19, 14:31
Posts: 883
Location: Italy
Role: Hobbyist
I have reworked the inheritances and some utilities around, and now it's quite stable, and in addition it now allows access to csv data as it were a table: you can access it based on row/column, only row, only column, and if you set it, also with a sort of index (first column) or field (first row)-based lookup.

I think I only need to completely comment the various classes, check how it works with the latest Duality Resource changes, and then I can start working on the nuGet integration part.

_________________
Come on Duality's Discord channel. We have cookies! :mrgreen:


Top
 Profile  
 
 Post subject: Re: [WIP] Data plugin
PostPosted: 2014/11/01, 19:08 
Site Admin
Site Admin
User avatar

Joined: 2013/05/11, 22:30
Posts: 2071
Location: Germany
Role: Professional
Sounds great :)

_________________
Blog | GitHub | Twitter (@Adams_Lair)


Top
 Profile  
 
PostPosted: 2014/11/08, 14:45 
Forum Addict
Forum Addict
User avatar

Joined: 2013/09/19, 14:31
Posts: 883
Location: Italy
Role: Hobbyist
:exclaim: :exclaim: RELEASED! See first post for details :exclaim: :exclaim:


For any issues or requests, just drop a message!

_________________
Come on Duality's Discord channel. We have cookies! :mrgreen:


Top
 Profile  
 
PostPosted: 2014/11/08, 18:50 
Site Admin
Site Admin
User avatar

Joined: 2013/05/11, 22:30
Posts: 2071
Location: Germany
Role: Professional
Awesome :D Just installed your plugin in a fresh Duality installation and it works. Also imported some XML files via dragdrop and marveled all those shiny new editor icons in the Project View.

Here are some things I noticed:

  • Your packages don't specify a title or icon. You should definitely consider adding both for future versions! Makes it more shiny and inviting to try them.
  • Your package IDs don't comply to the "standard" I'm trying to maintain for Duality packages. The scheme I'm trying to stick to is: AuthorName.Duality.Plugins.PluginName for core plugins and AuthorName.Duality.Editor.Plugins.PluginName for editor plugins. Of course, you don't have to use it, but it might spare our future selfs a bit of work and confusion if we all stick to the same scheme right from the beginning :)
    • When changing the ID, your new package version will be disconnected from the old one. That's fine when doing it now before it is widely used, but you'll have to manually unlist the old package on NuGet.org when viewing your packages while logged in.
    • Ideally, the package ID matches the main namespace of your code. Still, not a requirement, but also a scheme I'm trying to follow. Just a suggestion!
  • The category of the XmlSchema Resource is "File", but I think it wouldn't be too far off to just put it into "Data" besides the others as well. That way, all of your plugin additions would be nicely grouped.
  • I dragged EditorUserData.xml into the Project View, which is a tough test because that specific xml file does not comply to the xml standard - it contains multiple xml root nodes! I expected an error message in the log, or something similar, but unfortunately, the whole editor crashed :( This is the last log entry I got:
    Code:
    [Edit] ERROR:   Unexpected XML declaration. The XML declaration must be the first node in the document, and no white space characters are allowed to appear before it. Line 62, position 3.

    I'm pretty sure there also was an unhandled exception, but apparently it somehow didn't make it into the global handler for them and thus wasn't logged.
  • I edited AdamsLair.Winforms.xml after successfully importing it, expecting a hot-reload, but the editor crashed here as well.. again, no exception log :/
  • Nice work on editor integration! User-friendly icons, dragdrop, .. just like the core stuff :)
  • Also, the overall project structure in your GitHub repo nicely resembles the one I use for plugins.. which is a very good thing, because I would guess that's a big plus for other Duality developers trying to get into your plugin code :)

Hope this feedback was helpful. I'll look out for updates! :)

_________________
Blog | GitHub | Twitter (@Adams_Lair)


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 18 posts ]  Go to page 1, 2  Next

All times are UTC + 1 hour [ DST ]


Who is online

Users browsing this forum: No registered users and 2 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum

Jump to:  
cron
Powered by phpBB® Forum Software © phpBB Group