User:Skyfeather/VT OCR

This page contains a list of functions for reading text off the screen via Veggietales.

 void srStripScreen()
 void srStripRegion(int x0, int y0, int w, int h)
 region srFindFirstTextRegion()
 region srFindNextTextRegion()
 parse[] srParseTextRegion(int x0, int y0, int w, int h)
 region srFindInvRegion()
 region srFindChatRegion()
 void srTrainTextReader(int x, int y, char *c)

Objects introduced by this modification are:

 parse: parse[0] = x, parse[1] = y, parse[2] = text
 region: region[0] = x, region[1] = y, region[2] = width, region[3] = height

Parse objects include all the text on a given line within a text region.

lua Library Functions

I strongly recommend not using the above functions, and instead rely on the library functions added to screen_reader_common.inc. You need to have called srReadScreen() before most of these functions, unless otherwise listed. All functions return nil if nothing is found.

Returns an array of all text regions, except for inventory and chat regions. Regions are reduced in size on the right such that they don't include the thumbtack.

 region[] findAllTextRegions()

Returns the first parse object which includes the string s. findText("Wood") will find "Wood (494)" for example. Case sensitive.

 parse findText(String s)

Same as findText, but returns all parse objects, not just the first.

 parse findAllText(String s)

Strips a region of background color. Should never need if you use other library functions.

 void stripRegion(region r)

Returns an array of all text in a region

 parse[] parseRegion(region r)

same as findText, but only within a specific region.

 parse findTextInRegion(region r, String t)

same as findText, but looks inside a parse[]

 parse findTextInParse(parse[] p, String t)

Returns an array of regions that contain the text searched for. Useful for things like finding all Chemistry Lab windows.

 region[] findAllRegionsWithText(String t)

Parses chat and returns a parse[] such that each element contains all the text from a given chat phrase. Does this by looking for timestamps, if they're disabled, won't work.

 parse[] getChatText()

Returns a parse containing every line of chat on the screen, not including chat or inventory regions.

 parse[] getAllText()

Continues checking the screen until the searched for text appears. Does not return until the text is found, and does call checkBreak so that the loop can be interrupted. Do not have to call srReadScreen() first. Useful for checking whether button clicks go through and a new menu appears.

 parse waitForText(String t)

Does the same as above, but returns once the text is no longer on the screen. Useful for checking whether button clicks go through that close menus.

 void waitForNoText(String t)

same as waitForText, but returns the region instead.

 region waitForRegionWithText(String t)

same as WaitForText, but searches a specific region.

 parse waitForTextInRegion(Region r, String t)

Clicks at the location of a parse object, +20 x pixels, +7 y pixels.

 void clickText(parse p)

General parse process

The parsing mechanism works by finding what appears to be lines of text in a region by looking at horizontal lines and trying to find the horizontal rule that the text is aligned upon. It checks whether 1) The line contains mostly black pixels 2) The line below doesn't contain too many black pixels, and 3) the space 4 pixels below contain 0 black pixels. This can introduce some errors if trying to read too many consecutive characters that don't have any black pixels at the bottom part of the line, such as * or =. However, there is a hack in place to read consecutive +'s and -'s such as "Spice (++++)". Additionally, #3 is the reason that it has problems with vertical bars. It's also possible that it can have problems with lines containing too many characters that hang below the horizontal rule, such as p, q, and especially ( and ).

Pictures like the thumbtack can also make it difficult to parse text in some region, and the library function findAllTextRegions() reduces the size of regions on the right to not include the thumbtack. You may need to make adjustments if the window doesn't have a thumbtack, and has right-aligned text.

Known Issues and Miscellaneous

• Regions with vertical lines don't read correctly unless you shorten the size of the region. An example of this is the Browse menu in a chest. The best way to handle these is usually to find the region (with findRegionWithText), and then decrease the size of the region manually, by increasing region[0], and decreasing region[2].

• Some characters don't parse well, and intentionally aren't added to the list of known characters. _ and | specifically. _ currently introduces spurious l's as well.

• Colored text currently doesn't parse well, and the only supported colored text is the blue text in main. I recommend continuing to use the image search for checking the state of stat timers.

• If part of the edge of a text region is covered up, then the region won't be detected, and the text won't be parsed.

• Because regions are just rectangles, even if a region is covered up, if you call findTextInRegion, it will still parse as much text in the region as it can, sometimes even if part of it has become covered up since.

• If region B is completely contained within region A, then the text in region B *could* get parsed twice, but not necessarily.