VeggieTales Scripting Reference

This is documentation for editing scripts. For general usage and macro information, see Jimbly's Macros

This reference is now included in the VeggieTales installer under the file name ScriptingReference.txt. This wiki page last updated from VeggieTales version 1.77.

General Philosophy

Most of the existing macros and the macroing interface are all built around a few primary goals:

• No keyboard interaction. While a macro is running, the user should be able to freely chat with their friends, navigate between guild tabs, etc.
• Image recognition over offsets. Where possible, read and scrape the screen to find the exact image we want to click on instead of hardcoding pixel offsets that might change based on resolution or other factors. The downside of this is that it is more susceptible to minor changes in ATITD causing the macros to work.
• Status and prompts. Friendly prompts about what values are desired are easy to add, and much better than telling someone to edit the script to handle the values they desire.

API

The functions prefixed with "ls" (short for "LUA script", which makes more sense in the C++ code, as opposed to the actual macros where *everything* is lua script...) generally operate on the VeggieTales window, so lsGetWindowSize() gets the width and height of the VeggieTales window.

The functions prefixed with "sr" (short for Screen Reader) generally operate on the ATITD window, so srGetWindowSize() gets the width and height of the ATITD window.

The functions with neither an sr or ls prefix are lua-script library functions.

• Note that you *MUST* call srReadScreen to update the screen before calling srFindImageInRange etc.

Data Structures

• Vec2 (or Point) is a table where point[0] = x and point[1] = y representing a screen coordinate or size
• Region object is a table where region[0] = x0, region[1] = y0, region[2] = width, region[3] = height
• Parse object is a table where parse[0] = x0, parse[1] = y0, parse[2] = text
• [l,t,r,b] is a bounds table where bounds[0] = left, bounds[1] = top, bounds[2] = right, and bouds[3] = bottom
• Box is a table representing a rectangular region with fields (x, y, width, height, left, top, right, bottom)

Function List

General

void lsSetCaptureWindow()

Attaches screen-reader/clicking functions to the window under the mouse cursor. Call it once at the very beginning of script execution.

Use askForWindow() instead in most cases.

void lsSleep(int ms)

Pauses execution for a set period.

For long waits, use sleepWithStatus() or sleepWithStatusPause() in ui_common.inc

string lsScriptName()

Returns the name of the currently running script.

Sending Mouse Input

void srClickMouse(int x, int y, bool bRightClick)

Moves mouse cursor (x, y) and sends a click event.

Use safeClick() from common_click.inc instead in most cases.

void srClickMouseNoMove(int x, int y, bool bRightClick)

Sends a click event at (x, y) without moving the mouse cursor.

Use safeClick() from common_click.inc instead in most cases.

void srDrag(int x0, int y0, int x1, int y1, bool rightClick)

Drags the mouse from (x0, y0) to (x1, y1), moving the mouse cursor.

Use safeDrag() from common_click.inc instead in most cases.

void srSetMousePos(int x, int y)

Moves the mouse cursor to (x, y).

Usually call only immediately after safeBegin() from common_click.inc to minimize interference by the user.

void srMouseDown(int x, int y, bool rightClick)

void srMouseUp(int x, int y, bool rightClick)

Send mouse down/up event at position (x, y).

Usually call only immediately after safeBegin() to minimize interference by the user.

x,y srMousePos()

Get the current mouse position.

Use getMousePos() instead as it returns a standard Vec2 (Point) rather than two integers.

Sending Keyboard Input

In general, avoid when possible because chat state and settings may interfere with it.

void srKeyEvent(const char *s)

Sends keyboard events (up and down) for each character in the string.

void srLeftArrow()

void srRightArrow()

void srUpArrow()

void srDownArrow()

Sends arrow keypress events.

void srKeyDown(int key)

Send a key down event by key code

void srKeyUp(int key)

Send a key up event by key code

void srKeyEvent2(const char *s)

void srLeftArrow2()

void srRightArrow2()

void srUpArrow2()

void srDownArrow2()

void srKeyDown2(int key)

void srKeyUp2(int key)

Same, but uses the Windows SendInput API, see notes below.

Screen Reading

Vec2 srGetWindowSize()

Returns the capture window size. This is automated program's (e.g. ATITD's) window.

Call only after lsSetCaptureWindow() or askForWindow() have been called.

Vec2 srImageSize(const char *fn)

Returns the size in pixels of the image at a given path.

void srReadScreen()

Refresh the screen buffer. Functions which search the screen use the screen as it was when this function was last called.

Make sure to call this before any screen searching and any time you think the screen may have changed in a way important to your script.

void srMakeImage(const char *name, int x, int y, int w, int h)

Create a new image 'file', referenced later by the 'name' at the given path based on the pixels found on part of the atitd screen. srMakeImage does not make a file on disk, it makes an "image" to be used for srFindImage, or displayed with srShowImageDebug (see multi_click.lua). The image "name" can be referenced in the above functions.

Vec2 srFindImage(const char *fn, int tol)

Search the screen for a match on a particular image and return the coordinates of its upper-left corner. lsSetCaptureWindow() or askForWindow() must have been called. This function searches the screen as it was when srReadScreen() was last called. This function finds the top-most then left-most match.

To find multiple matches, see findAllImages().

void srShowImageDebug(const char *fn, float x, float y, float z, float scale)

Show an image, such as one captured with srMakeImage, in the VT window, useful for debugging, or feedback in things like multi_click.lua.

Vec2 srFindImageInRange(const char *fn, int x0, int y0, int w, int h, int tol)

Like srFindImage(), but only searches a portion of the screen.

This can be much faster than searching the entire screen.

int srReadPixel(int x, int y)

Get the pixel value at a particular location in RGBA format (to get RGB format, divide by 256 rounding down). This function reads directly from the screen, so is always up to date, and is much cheaper than refreshing the entire screen buffer, but much more expensive than reading a pixel from the captured buffer.

int srReadPixelFromBuffer(int x, int y)

Similar to the above function but fetches the pixel at a location as of the last srReadScreen() call.

void srSaveLastReadScreen(const char *fn)

Save whole screen buffer to file, useful for debugging.

Vec2 srFindNearestPixel(lua_State *L, int color, int x, int y, bool checkRgb, bool checkHue, int radius, int rgbTol, int hueTol)

Returns the position of a pixel within the specified circular area which either matches the tolerance by RGB or Hue.

Tolerances are squared difference sums (so, range of 0 to 255*255^3, recommended around 5000, maybe less for Hue).

void lsShowScreengrab(const char *_color)

Draws the captured screen in the VT window.

Vec2[] lsAnalyzePapyrus(int minsize, int maxsize, int debug, int relevant_radius)

Returns an array of positions on screen suspected to be papyrus.

Vec2[] lsAnalyzeSilt(int minsize, int maxsize, int debug, int relevant_radius)

Returns an array of positions on screen suspected to be silt.

[l,t,r,b] srGetWindowBorders(int x, int y)

Returns the bounds of the in-game window surrounding (x, y). It may return odd results if (x, y) is not inside an in-game window or if the window that (x, y) is over is behind other windows. The result looks at the screen as of the last time srReadScreen() was called.

Use getWindowBorders() instead as it returns a standard Box rather than a bounds.

[r,g,b] lsColorComponent(color)

Converts a screen read color to RGB components

Text Parsing (raw API, ATITD-specific)

 It is better to use the library functions below as they provide a higher level interface than the raw API.

region srFindInvRegion()

Returns the text region enclosing the inventory/skills area.

Use findInventoryRegion() instead as it returns a standard Box rather than a region.

void srStripScreen()

void srStripRegion(int x0, int y0, int w, int h)

Removes the background color from a particular area. Used internally by the text library below.

region srFindFirstTextRegion()

Finds the first text region on the screen and resets the internal iterator. Used internally by the text library below.

region srFindNextTextRegion()

Returns the next region based on the internal iterator or nil. Used internally by the text library below.

parse[] srParseTextRegion(int x0, int y0, int w, int h)

Finds all text in a particular part of the screen. Used internally by the text library below.

region srFindChatRegion()

Returns the region enclosing the chat area.

Use findChat Region() instead as it returns a standard Bo rather than a region.

void srTrainTextReader(int x, int y, const char *c)

Used internally by the text library below.

Text Parsing (library, ATITD-specific)

See Skyfeather's OCR Notes at http://www.atitd.org/wiki/tale6/User:Skyfeather/VT_OCR Skyfeather's OCR Notes

 region[] findAllTextRegions()
 parse findText(String s)
 void stripRegion(region r)
 parse[] parseRegion(region r)
 parse findTextInRegion(region r, String t)
 region[] findAllRegionsWithText(String t)
 parse waitForText(String t)
 void waitForNoText(String t)
 region waitForRegionWithText(String t)
 parse waitForTextInRegion(Region r, String t)
 void clickText(parse p)
 parse[] getChatText()
 parse[] getAllText()
 parse findTextInParse(parse[] p, String t)

User Interface

int lsScreenX

int lsScreenY

Vec2 lsGetWindowSize()

Width/height of the VT interface window.

Vec2 lsMouseClick(int x, int y, int w, int h, int button)

Returns the coordinates of and captures a click in the specified portion of the VT window.

bool lsMouseIsDown(int button)

Returns true if the given mouse button is currently being pressed by the user. 1 = Left, 2 = Middle, 3 = Right.

void lsPrintln(const char *s)

Print a string to the console. You can open the console from the VT main menu. Used for debugging.

bool lsShiftHeld()

bool lsAltHeld()

bool lsControlHeld()

Returns true if the player is currently holding the given button. Make sure to check for the user releasing the button between taking actions based on the button state.

void lsPlaySound(const char *sound)

Play an audio file at the given path.

int lsPrint(int x, int y, int z, float xsc, float ysc, const char *color, const char *str)

Print text to the VT interface. Displayed when lsDoFrame() is called.

void lsFontShadow(int on)

Switches to a shadowed font, this can be useful if displaying text over an image.

int lsMessageBox(const char *title, const char *msg, int buttons)

Displays a message box with an OK button (1, default), Yes (2), No (4), or Cancel (8) buttons, and returns which button is clicked. Combine the bits to specify a set of buttons (e.g. 2+4=6 will show both "Yes" and "No" buttons).

int lsPrintWrapped(int x, int y, int z, int w, float xsc, float ysc, const char *color, const char *str)

Print wrapped text on the VT interface. Displayed when lsDoFrame() is called.

void lsDisplaySystemSprite(int tile, int x, int y, int z, float w, float h, const char *color)

Display image on the VT interface. tile = 1 is just a solid white box, which can be used, when combined with a color, to draw a colored box of the requested size.

bool lsButtonText(int x, int y, int z, int w, const char *color, const char *txt)

Display a text button on the VT interface and return true if it was clicked. Displayed when lsDoFrame() is called.

bool lsButtonImg(int x, int y, int z, float scale, const char *color, const char *filename)

Display an image button on the VT interface and return true if it was clicked. Displayed when lsDoFrame() is called.

bool lsCheckBox(int x, int y, int z, const char *color, const char *txt, bool value)

Display a check box that the user can toggle. Returns the current value of the check box. Displayed when lsDoFrame() is called.

int lsDropdown(const char *key, int x, int y, int z, int w, int val)

Display a dropdown menu of items the user can select. Returns the index currently selected. Displayed when lsDoFrame() is called.

void lsScrollAreaBegin(const char *key, int x, int y, int z, int w, int h)

Display a scrollable area on the VT interface. All UI done until lsScrollAreaEnd() will be within this scrollable area.

void lsScrollAreaEnd(int h)

End a scrollable area

void lsSetCamera(float x0, float y0, float x1, float y1)

Define the coordinate space used for the VT window. Can be used to zoom out the interface globally.

void lsDoFrame()

Update the screen with the set of buttons/text/controls invoked since the last lsDoFrame(). lsDoFrame() called twice in a row will produce a blank screen.

void lsTopmost(int value)

Set whether the VT window is "topmost" (appearing on top of all other windows). Sometimes this setting can get lost, or for specific scripts it may be needed to turn it off temporarily.

{bool done, selection} lsEditBox(const char *key, int x, int y, int z, int w, int h, float xsc, float ysc, const char *color, const char *def)

Provide an edit box for the user to enter text into. selection returns the current contents of the box. Done is set to true when the use presses enter.

int lsGetTimer()

Returns the current time in milliseconds.

const char *lsClipboardGet()

Returns the current contents of the clipboard.

void lsClipboardSet(const char *str)

Sets the contents of the clipboard.

luaSocket library

This is a 3rd party Lua module that is included in VeggieTales to allow scripts to communicate, using HTTP, with external services.

Full reference can be found at http://w3.impa.br/~diego/software/luasocket/reference.html

Simple usage:

 local http = require("socket.http");
 local body, code, headers = http.request("http://www.google.com");

Network Server

void lsServerListen()

Listens on port 3007 for GlovNet connections. Client code to connect to this can be found in libUtil in the same repository as the VeggieTales source code.

{ip, client, type[, cmd, data]} lsServerGetNextEvent(int timeout)

Polls for incoming network events. Type is "connect", "disconnect", or "data". If type is "data", then "cmd" and "data" will also be filled in with the command and payload.

void lsServerSend(int client, int cmd, const char *data)

Send a message to a client (client ids can be recorded from their "connect" message handled above).

Specific reference

User Interface

The user interface is what is called "immediate mode" rather than "retained mode". This means that in order to present a consistent user interface to the user, you need to wrap a bunch of controls in a loop:

 while not done do
   -- Code to print text to the screen and provide controls the user can interact with
   lsDoFrame();
   lsSleep(tick_delay);
   checkBreak();
   -- Code to act based on button clicks, etc. and set done if necessary.
 end

lsDoFrame() actually displays content in the VT window while adding a short sleep prevents the loop from consuming too many system resources. The code to actually implement user actions should be after the lsDoFrame() because if not, there will be graphical glitches if the action tries to update the VT screen. Jimbly: Though it is true that you can get some graphical glitches if you're responding to things immediately, delaying the actions based on button clicks until the end of the loop a) adds some actual delay time, and b) loses a lot of the simplicity and readability of an immediate-mode user interface.

checkBreak() will exit the program if the user is pressing both shift and control. In any indefinite loop you make, this should be included to allow the user to abort your macro without waiting.

Two library functions are provided to help in the common cases of providing a user interface.

statusScreen() allows you to post a status message to the user while providing an explicit button allowing the user to quit the script. You can use this periodically in long-running tasks to make sure that the user knows what is going on and can abort the script at any time.

sleepWithStatus() allows you to pause for long periods of time while giving feedback to the user and allowing them to abort the script in the middle.

Library Functions

Global Variables

tick_delay

The amount of time to wait between iterations in a polling loop (in ms). Default: 10

click_delay

The amount of time to wait between clicks when making many clicks at once (in ms). Default: 50

allow_pause

Whether to allow the user to pause the macro or not. Affects checkBreak(), statusScreen(), sleepWithStatus(). Set to false if your code requires delicate timing. Default: true

quit_message

Standard message to print to the user if they explicitly abort the macro.

General Functions

void askForWindow(String t)

Pause and tell the user to press shift over the target window. Should be run only once and near the beginning of the macro.

Vec2 makePoint(x, y)

Make a Vec2 (or Point) data structure from a pair of coordinates.

Box makeBox(x, y, width, height)

Make a box data structure which represents a portion of the screen.

Vec2[] findAllImages(String imgName, int tol)

Return a list of all occurrences of an image on the screen

Mouse

void safeBegin()

Performs various checks to make sure that we don't interfere with the user. This takes a small amount of additional time and should be used before performing mouse actions using the standard API.

void safeClick(x, y, rightClick)

Clicks without moving the mouse after checking to make sure we don't interfere with the user.

bool safeDrag(sourceX, sourceY, destX, destY, timeout)

Drags from source to destination after checking to make sure we don't interfere with the user. Waits until drag completes by examining the destination pixel for change. Timeout (in ms) is the maximum amount of time it will wait. Returns true on success, false on timeout.

void clickAllPoints(points, offsetX, offsetY, rightClick)

Click every point in a list in sequence adding (offsetX, offsetY) to each click position. (offsetX, offsetY) defaults to (5, 5).

int clickAllImages(image, offsetX, offsetY, rightClick, tol)

Click all matching images, but add (offsetX, offsetY) to each click position (offset defaults to 5, 5). Returns the number of clicks performed.

bool drawWater()

Attempts to draw water by first looking for a rain barrel, second trying to find an aqueduct, and third attempting to find the fill water icon. Returns true on success, false if no water source was found.

Vec2 getMousePos()

Returns the current location of the mouse.

Automating mouse clicks (low level details)

srClickMouseNoMove() sends a mouse down and up event (WM_LBUTTONDOWN/WM_LBUTTONUP) at the specified coordinates, to the automated window, without moving the cursor. For games / programs, like ATITD, that listen to the coordinates of a mouse click (as opposed to polling the mouse position and processing a click whever the mouse last was), when only a click is required, this works ideally, because it does not completely steal control of the user's mouse while the macro is running. Additionally it works while the automated program's window is not even in focus! This also means that clicks are never accidentally sent to the wrong program. However, if a use us moving the mouse, or dragging something else, while this function is called, things won't work how either the script or the user expect (probably will release the user's click at the location of the script's click, or something like that).

However, if the program does not listen to mouse events in that particular way, or the operation involved requires dragging, then the mouse cursor actually needs to be moved to the appropriate location on screen. srClickMouse() uses a different API ("mouse_event") and simulates moving the mouse, waiting a moment, sending a mouse down, waiting a moment, sending a mouse up, and then waiting a moment before returning control to the script. The times needed here vary based on program and performance, generally needs to wait long enough for a "frame" of the program being automated to process, so this time may need to be adjusted based on system performance and framerate. For example, for ATITD I found the delays of 50,15,15 work well, and for Atom Zombie Smasher I had to use values of 50,100,50 for a script to be reliable. This method requires the window to be in focus, and if the window is not in focus might lose the click, since, for some programs, the first click just activates the window and is otherwise ignored.

In general, stick with srClickMouseNoMove, unless the program doesn't seem to respond to this at all, then use srClickMouse. There are more complicated things you can do to make clicks safer, check out safeClick() in common_click.inc in the scripts for ATITD.

Automating keyboard input (low level details)

There are two sets of keyboard input functions, one suffixed with "2". The original set use window messages send directly to the automated program. Like srClickMouseNoMove() above, this is ideal since it does not require the window to be in focus, does not interrupt the user as much, and cannot send keyboard input to the wrong program. However, not all programs respond to these events, and for those that don't, you can use the "2" varieties which instead make use of the "SendInput" API which should work in all programs.

void srKeyEvent(const char *s)

Sends a character or sequence of characters to the captured window. Example:

 loadfile("luaScripts/screen_reader_common.inc")();
 function doit()
   askForWindow();
   lsSleep(250); -- wait for shift release to make it to the window
   srKeyEvent('username\tpassword');
   lsSleep(250); -- wait a moment before submitting so we can see it work, that's more fun!
   srKeyEvent('\n');
 end

This will send the string "username" followed by a tab (\t) and then the string "password" and then enter (\n).

void srKeyDown(int key)

srKeyDown() sends a key down message. When most keyboard keys are pressed, they trigger repeating messages. To simulate this, call srKeyDown() repeatedly with the same key. srKeyDown() will recognize multiple calls prior to the srKeyUp() call and send appropriate repeating key messages. Multiple keys may be down at once.

void srKeyUp(int key)

srKeyUp() sends a key up message, releasing a key pressed with srKeyDown(). To release all pressed keys, send the special key code -1.

Misc Notes

Explanation of tol (tolerance) value in srFindImage() and srFindImageInRange()

Me: Can you explain how the tolerance range works srFindImage("waterw.png", 1) . I've seen it range from 1 to 5000, how does the range work?

Jimbly: srFindImage looks for a block of pixels similar to the specified image, where no pixel has a summed squared difference larger than the tolerance. A tolerance of 0 or 1 would require an exact pixel for pixel match, whereas a larger tolerance would allow for mild variation (most often the background behind the text is different, since ATITD has a textured background behind windows).

Jimbly: An example difference is if the provided image has an RGB of 100, 150, 200, and the screen read had an RGB value of 110, 145, 180, the summed squared difference is 10*10 + 5*5 + 20*20 = 525

Jimbly: Default tolerance, if none is specified is 4500, which seemed to always find a match, and never get a false positive. I seem to remember using 5000 everywhere, but just saw in the code it uses 4500 if none is specified

Jimbly: I've found things that use a low tolerance are much more likely to break (most common being if, say, another line of text is added to a window, the tolerance of 1 will certainly not find what its looking for anymore, where as a higher tolerance will find it even if it's lower on the screen (and has slightly different background behind the text).

Summary: Its probably safer to start out leaving the tolerance blank (default of 4500) unless you have a specific need to find something precise down to the last pixel. ie use srFindImage("Harvest.png"); instead of srFindImage("Harvest.png", 1); If you have a need to narrow down a search to be more precise, then start adding numbers to the tol value.

lsDoFrame()

This draws the VeggieTales UI, any queued up lsPrint, buttons, etc. So, if you want the user to see anything, it needs to be called. Just drawing text over and over in a loop won't ever show up on screen unless lsDoFrame(); is called.

Text Recongition Notes

See Skyfeather's OCR Notes at http://www.atitd.org/wiki/tale6/User:Skyfeather/VT_OCR Skyfeather's OCR Notes

Command Line Options

Run a specified script immediately: > VeggieTales.exe -run flax.lua Run thistles mode immediately: > VeggieTales.exe -thistles