VeggieTales Scripting Reference

This is documentation for editing scripts. For general usage and macro information, see Jimbly's Macros

This reference is now included in the VeggieTales installer under the file name ScriptingReference.txt. This wiki page last updated from VeggieTales version 1.60.

General Philosophy

Most of the existing macros and the macroing interface are all built around a few primary goals:

• No keyboard interaction. While a macro is running, the user should be able to freely chat with their friends, navigate between guild tabs, etc.
• Image recognition over offsets. Where possible, read and scrape the screen to find the exact image we want to click on instead of hardcoding pixel offsets that might change based on resolution or other factors. The downside of this is that it is more susceptible to minor changes in ATITD causing the macros to work.
• Status and prompts. Friendly prompts about what values are desired are easy to add, and much better than telling someone to edit the script to handle the values they desire.

API

The functions prefixed with "ls" (short for "LUA script", which makes more sense in the C++ code, as opposed to the actual macros where *everything* is lua script...) generally operate on the VeggieTales window, so lsGetWindowSize() gets the width and height of the VeggieTales window.

The functions prefixed with "sr" (short for Screen Reader) generally operate on the ATITD window, so srGetWindowSize() gets the width and height of the ATITD window.

The functions with neither an sr or ls prefix are lua-script library functions.

• Note that you *MUST* call srReadScreen to update the screen before calling srFindImageInRange etc.

Data Structures

 Vec2 (or Point) is a table where point[0] = x and point[1] = y representing a screen coordinate or size
 Region object is a table where region[0] = x0, region[1] = y0, region[2] = width, region[3] = height
 Parse object is a table where parse[0] = x0, parse[1] = y0, parse[2] = text
 [l,t,r,b] is a bounds table where bounds[0] = left, bounds[1] = top, bounds[2] = right, and bouds[3] = bottom
 Box is a table representing a rectangular region with fields (x, y, width, height, left, top, right, bottom)

Function List

General

void lsSetCaptureWindow()

Attaches screen-reader/clicking functions to the window under the mouse cursor. Call it once at the very beginning of script execution.

Use askForWindow() instead in most cases.

void lsSleep(int ms)

Pauses execution for a set period.

For long waits, use sleepWithStatus() or sleepWithStatusPause().

Sending Input

void srClickMouse(int x, int y, bool bRightClick)

Moves mouse cursor (x, y) and sends a click event.

Use safeClick() instead in most cases.

void srClickMouseNoMove(int x, int y, bool bRightClick)

Sends a click event at (x, y) without moving the mouse cursor.

Use safeClick() instead in most cases.

void srKeyEvent(const char *s)

Sends keyboard events.

Avoid when possible as it may interact badly with the player's chat.

void srLeftArrow()

void srRightArrow()

void srUpArrow()

void srDownArrow()

Sends arrow keypress events.

Avoid when possible because chat state and settings may interfere with it.

void srDrag(int x0, int y0, int x1, int y1, bool rightClick)

Drags the mouse from (x0, y0) to (x1, y1), moving the mouse cursor.

Use safeDrag() instead in most cases.

void srSetMousePos(int x, int y)

Moves the mouse cursor to (x, y).

Usually call only immediately after safeBegin() to minimize interference by the user.

void srMouseDown(int x, int y, bool rightClick)

void srMouseUp(int x, int y, bool rightClick)

Send mouse down/up event at position (x, y).

Usually call only immediately after safeBegin() to minimize interference by the user.

x, y srMousePos()

Get the current mouse position.

Use getMousePos() instead as it returns a standard Vec2 (Point) rather than two integers.

Screen Reading

Vec2 srGetWindowSize()

Returns the capture window size. This is generally the atitd window.

Call only after lsSetCaptureWindow() or askForWindow() have been called.

Vec2 srImageSize(const char *fn)

Returns the size in pixels of the image at a given path.

void srMakeImage(const char *name, int x, int y, int w, int h)

Create a new image 'file', referenced later by the 'name' at the given path based on the pixels found on part of the atitd screen. srMakeImage does not make a file on disk, it makes an "image" to be used for srFindImage, or displayed with srShowImageDebug (see multi_click.lua). The image "name" can be referenced in the above functions.

Vec2 srFindImage(const char *fn, int tol)

Search the screen for a match on a particular image and return the coordinates of its upper-left corner. lsSetCaptureWindow() or askForWindow() must have been called. This function searches the screen as it was when srReadScreen() was last called. This function finds the top-most then left-most match.

To find multiple matches, see findAllImages().

void srShowImageDebug(const char *fn, float x, float y, float z, float scale)

Show an image in the VT window?

Vec2 srFindImageInRange(const char *fn, int x0, int y0, int w, int h, int tol)

Like srFindImage(), but only searches a portion of the screen.

void srReadScreen()

Refresh the screen buffer. Functions which search the screen use the screen as it was when this function was last called.

Make sure to call this before any screen searching and any time you think the screen may have changed in a way important to your script.

int srReadPixel(int x, int y)

Get the pixel value at a particular location in RGBA format (to get RGB format, divide by 256 rounding down). This function always refreshes the screen buffer so it is expensive but timely.

int srReadPixelFromBuffer(int x, int y)

Similar to the above function but fetches the pixel at a location as of the last srReadScreen() call.

void srSaveLastReadScreen(const char *fn)

Save whole screen buffer to file?

void lsShowScreengrab(const char *_color)

Unknown

Vec2[] lsAnalyzePapyrus(int minsize, int maxsize, int debug, int relevant_radius)

Unknown

Vec2[] lsAnalyzeSilt(int minsize, int maxsize, int debug, int relevant_radius)

Unknown

[l,t,r,b] srGetWindowBorders(int x, int y)

Returns the bounds of the in-game window surrounding (x, y). It may return odd results if (x, y) is not inside an in-game window or if the window that (x, y) is over is behind other windows. The result looks at the screen as of the last time srReadScreen() was called.

Use getWindowBorders() instead as it returns a standard Box rather than a bounds.

Text Parsing

 It is better to use the library functions below as they provide a higher level interface than the raw API.

region srFindInvRegion()

Returns the text region enclosing the inventory/skills area.

Use findInventoryRegion() instead as it returns a standard Box rather than a region.

void srStripScreen()

void srStripRegion(int x0, int y0, int w, int h)

Removes the background color from a particular area. Used internally by the text library below.

region srFindFirstTextRegion()

Finds the first text region on the screen and resets the internal iterator. Used internally by the text library below.

region srFindNextTextRegion()

Returns the next region based on the internal iterator or nil. Used internally by the text library below.

parse[] srParseTextRegion(int x0, int y0, int w, int h)

Finds all text in a particular part of the screen. Used internally by the text library below.

region srFindChatRegion()

Returns the region enclosing the chat area.

Use findChat Region() instead as it returns a standard Bo rather than a region.

void srTrainTextReader(int x, int y, const char *c)

Used internally by the text library below.

User Interface

int lsScreenX

int lsScreenY

Vec2 lsGetWindowSize()

Width/height of the VT interface window.

Vec2 lsMouseClick(int x, int y, int w, int h, int button)

Unknown

bool lsMouseIsDown(int button)

Returns true if the given mouse button is currently being pressed by the user. 1 = Left, 2 = Middle, 3 = Right.

void lsPrintln(const char *s)

Print a string to the console. You can open the console from the VT main menu. Used for debugging.

bool lsShiftHeld()

bool lsAltHeld()

bool lsControlHeld()

Returns true if the player is currently holding the given button. Make sure to check for the user releasing the button between taking actions based on the button state.

void lsPlaySound(const char *sound)

Play an audio file at the given path.

int lsPrint(int x, int y, int z, float xsc, float ysc, const char *color, const char *str)

Print text to the VT interface. Displayed when lsDoFrame() is called.

void lsFontShadow(int on)

Unknown

int lsMessageBox(const char *title, const char *msg, int buttons)

Unknown

int lsPrintWrapped(int x, int y, int z, int w, float xsc, float ysc, const char *color, const char *str)

Print wrapped text on the VT interface. Displayed when lsDoFrame() is called.

void lsDisplaySystemSprite(int tile, int x, int y, int z, float w, float h, const char *color)

Display image on the VT interface?

bool lsButtonText(int x, int y, int z, int w, const char *color, const char *txt)

Display a text button on the VT interface and return true if it was clicked. Displayed when lsDoFrame() is called.

bool lsCheckBox(int x, int y, int z, const char *color, const char *txt, bool value)

Display a check box that the user can toggle. Returns the current value of the check box. Displayed when lsDoFrame() is called.

int lsDropdown(const char *key, int x, int y, int z, int w, int val)

Display a dropdown menu of items the user can select. Returns the index currently selected. Displayed when lsDoFrame() is called.

void lsScrollAreaBegin(const char *key, int x, int y, int z, int w, int h)

Unknown

void lsScrollAreaEnd(int h)

Unknown

void lsSetCamera(float x0, float y0, float x1, float y1)

Unknown

void lsDoFrame()

Update the screen with the set of buttons/text/controls invoked since the last lsDoFrame(). lsDoFrame() called twice in a row will produce a blank screen.

void lsTopmost(int value)

Unknown

{bool done, selection} lsEditBox(const char *key, int x, int y, int z, int w, int h, float xsc, float ysc, const char *color, const char *def)

Provide an edit box for the user to enter text into. selection returns the current contents of the box. Not sure what the purpose of done is.

int lsGetTimer()

Returns the current time in milliseconds.

const char *lsClipboardGet()

Returns the current contents of the clipboard.

void lsClipboardSet(const char *str)

Sets the contents of the clipboard.

Specific reference

void srKeyEvent(const char *s)

Sends a character or sequence of characters to the captured window. Example:

 loadfile("luaScripts/screen_reader_common.inc")();
 function doit()
   askForWindow();
   lsSleep(250); -- wait for shift release to make it to the window
   srKeyEvent('username\tpassword');
   lsSleep(250); -- wait a moment before submitting so we can see it work, that's more fun!
   srKeyEvent('\n');
 end

This will send the string "username" followed by a tab (\t) and then the string "password" and then enter (\n).

User Interface

The user interface is active rather than event driven. This means that in order to present a consistent user interface to the user, you need to wrap a bunch of controls in a loop:

while not done do

 -- Code to print text to the screen and provide controls the user can interact with
 lsDoFrame();
 lsSleep(tick_delay);
 checkBreak();
 -- Code to act based on button clicks, etc. and set done if necessary.

end

lsDoFrame() actually displays content in the VT window while adding a short sleep prevents the loop from consuming too many system resources. The code to actually implement user actions should be after the lsDoFrame() because if not, there will be graphical glitches if the action tries to update the VT screen.

checkBreak() will exit the program if the user is pressing both shift and control. In any indefinite loop you make, this should be included to allow the user to abort your macro without waiting.

Two library functions are provided to help in the common cases of providing a user interface.

statusScreen() allows you to post a status message to the user while providing an explicit button allowing the user to quit the script. You can use this periodically in long-running tasks to make sure that the user knows what is going on and can abort the script at any time.

sleepWithStatus() allows you to pause for long periods of time while giving feedback to the user and allowing them to abort the script in the middle.

Library Functions

Global Variables

tick_delay

The amount of time to wait between iterations in a polling loop (in ms). Default: 10

click_delay

The amount of time to wait between clicks when making many clicks at once (in ms). Default: 50

allow_pause

Whether to allow the user to pause the macro or not. Affects checkBreak(), statusScreen(), sleepWithStatus(). Set to false if your code requires delicate timing. Default: true

quit_message

Standard message to print to the user if they explicitly abort the macro.

General Functions

void askForWindow(String t)

Pause and tell the user to press shift over the target window. Should be run only once and near the beginning of the macro.

Vec2 makePoint(x, y)

Make a Vec2 (or Point) data structure from a pair of coordinates.

Box makeBox(x, y, width, height)

Make a box data structure which represents a portion of the screen.

Mouse

void safeBegin()

Performs various checks to make sure that we don't interfere with the user. This takes a small amount of additional time and should be used before performing mouse actions using the standard API.

void safeClick(x, y, rightClick)

Clicks without moving the mouse after checking to make sure we don't interfere with the user.

bool safeDrag(sourceX, sourceY, destX, destY, timeout)

Drags from source to destination after checking to make sure we don't interfere with the user. Waits until drag completes by examining the destination pixel for change. Timeout (in ms) is the maximum amount of time it will wait. Returns true on success, false on timeout.

void clickAllPoints(points, offsetX, offsetY, rightClick)

Click every point in a list in sequence adding (offsetX, offsetY) to each click position. (offsetX, offsetY) defaults to (5, 5).

int clickAllImages(image, offsetX, offsetY, rightClick, tol)

Click all matching images, but add (offsetX, offsetY) to each click position (offset defaults to 5, 5). Returns the number of clicks performed.

bool drawWater()

Attempts to draw water by first looking for a rain barrel, second trying to find an aqueduct, and third attempting to find the fill water icon. Returns true on success, false if no water source was found.

Vec2 getMousePos()

Returns the current location of the mouse.

Misc Notes

Explanation of tol (tolerance) value in srFindImage() and srFindImageInRange()

Me: Can you explain how the tolerance range works srFindImage("waterw.png", 1) . I've seen it range from 1 to 5000, how does the range work?

Jimbly: srFindImage looks for a block of pixels similar to the specified image, where no pixel has a summed squared difference larger than the tolerance. A tolerance of 0 or 1 would require an exact pixel for pixel match, whereas a larger tolerance would allow for mild variation (most often the background behind the text is different, since ATITD has a textured background behind windows).

Jimbly: An example difference is if the provided image has an RGB of 100, 150, 200, and the screen read had an RGB value of 110, 145, 180, the summed squared difference is 10*10 + 5*5 + 20*20 = 525

Jimbly: Default tolerance, if none is specified is 4500, which seemed to always find a match, and never get a false positive. I seem to remember using 5000 everywhere, but just saw in the code it uses 4500 if none is specified

Jimbly: I've found things that use a low tolerance are much more likely to break (most common being if, say, another line of text is added to a window, the tolerance of 1 will certainly not find what its looking for anymore, where as a higher tolerance will find it even if it's lower on the screen (and has slightly different background behind the text).

Summary: Its probably safer to start out leaving the tolerance blank (default of 4500) unless you have a specific need to find something precise down to the last pixel. ie use srFindImage("Harvest.png"); instead of srFindImage("Harvest.png", 1); If you have a need to narrow down a search to be more precise, then start adding numbers to the tol value.

lsDoFrame()

This draws the VeggieTales UI, any queued up lsPrint, buttons, etc. So, if you want the user to see anything, it needs to be called. Just drawing text over and over in a loop won't ever show up on screen unless lsDoFrame(); is called.

Text Recongition Notes

See Skyfeather's OCR Notes